We study the dynamical behavior of a single degree of freedom mechanical system with a particle damper. The particle (granular) damping was optimized for the primary system operating condition by using an appropriate gap size for a prismatic enclosure. The particles absorb the kinetic energy of the vibrating structure and convert it into heat through the inelastic collisions and friction. This results in a highly nonlinear mechanical system. Considering linear signal analysis, state space reconstruction, Poincaré sections and the determination of maximal Lyapunov exponents, the motion of the granular system inside the enclosure is characterized for a wide frequency range. With the excitation frequency as control parameter, either regular or chaotic motion of the granular bed is found and the influence on the damping is analyzed.