I use a Pi Zero with ws2812b, with an arduino nano over USB. That works flawlessly.
I had no luck on the direct connection either, if I recall right.
Oh that’s an interesting idea. How are you using the raspberry pi and arduino? Is the pi interfacing with the arduino and the arduino controlling the LED?