Many extensions of the Standard Model posit the existence of heavy particles with long lifetimes. In this letter, results are presented of a search for such particles, which decay at a significant distance from their production point, using a final state containing charged hadrons and an associated muon. This analysis uses a data sample of proton-proton collisions at a center-of-mass energy of 7 TeV corresponding to an integrated luminosity of 4.4fb-1 collected in 2011 by the ATLAS detector operating at the Large Hadron Collider. Results are interpreted in the context of R-Parity violating supersymmetric scenarios. No events in the signal region are observed and limits are set on the supersymmetry production cross-section as a function of the neutralino lifetime. To allow these limits to be used in a variety of models, they are presented for a range of squark and neutralino masses.