A strange avg~800 DQN agent for Gymnasium Car-Racing v3 Randomize = True Environment

Hi everyone!

I ran a side project to challenge myself (and help me learn reinforcement learning).

“How far can a Deep Q-Network (DQN) go on CarRacing-v3, with domain_randomize=True?”

Well, it turns out… weird….

I trained a DQN agent using only Keras (no PPO, no Actor-Critic), and it consistently scores around 800+ avg over 100 episodes, sometimes peaking above 900.

All of this was trained with domain_randomize=True enabled.

All of this is implemented in pure Keras, I don’t use PPO, but I think the result is weird…

I could not 100% believe in this one, but I did not find other open-source agents (some agents are v2 or v1). I could not make a comparison…

That said, I still feel it’s a bit *weird*.

I haven’t seen many open-source DQN agents for v3 with randomization, so I’m not sure if I made a mistake or accidentally stumbled into something interesting.

A friend encouraged me to share it here and get some feedback.

I put this agent on GitHub…GitHub repo (with notebook, GIFs, logs):
https://github.com/AeneasWeiChiHsu/CarRacing-v3-DQN-

In my plan, I made some choices and left some reasons (check the readme, but it is not very clear how the agent learnt it)…It is weird for me.

A brief tech note:
Some design choices:

– Frame stacking (96x96x12)

– Residual CNN blocks + multiple branches

– Multi-head Q-networks mimicking an ensemble

– Dropout-based exploration instead of noisyNet

– Basic dueling, double Q, prioritized replay

– Reward shaping (I just punished “do nothing” actions)

It’s not a polished paper-ready repo, but it’s modular, commented, and runnable on local machines (even on my M2 MacBook Air).

If you find anything off — or oddly weird — I’d love to know.

Thanks for reading!

(feedback welcome — and yes, this is my first time posting here 😅

And I want to make new friends here. We can study RL together!!!