Jittering and wobbling with specific custom video modes

[Shogun]

Hey all I have been experiencing this issue off and on while messing with video modes while trying to get things dialed in with my PC Monitor (Dell Trinitron D1626HT) at 720x400 in the AO486 core. While experimenting with various modes I notice some they caused the screen to shake really fast at this resolution. I tried to pinpoint which specific setting in the modeline was the culprit by testing a more common resolution at 640x480. After testing some mode lines I found the following results:

640,24,64,88,480,3,4,20,35000 (fast shaking)
640,56,56,80,480,1,3,25,36000 (fast shaking)
640,16,96,48,480,10,2,33,25175 (works without any issues)
640,16,64,80,480,3,4,13,23750 (slow wobble)

Can anyone decipher the results above to determine which specific setting has an effect on this jittering/wobbling thats going on? I need to get a non jittering resolution for 720x400 but so far everything I tested had this issue.

720x400 modes tested that had this issue:
720,36,72,108,400,1,3,42,35500
720,24,72,96,400,3,10,10,32750
720,48,32,80,400,3,10,6,31250

[Shogun]

Updating with a video of the issue. I am unable to find a modeline at 720x400 that doesn't exhibit this issue. The best one I've found is:

Code: Select all

video_mode=720,36,72,108,400,1,3,42,35500

This has the least perceptible flickering/jitter. The 640x480 wobble affect is shown in the video just to illustrate the differences between the two.

Video of issue here

Code: Select all

720x400 modelines tested:
720	36	72	108	400	1	3	42	35500 (Least perceptible shaking)
720	24	72	96	400	3	10	10	32750
720	48	32	80	400	3	10	6	31250
720	24	64	88	400	3	10	6	22250 (Out of scan range)
720	24	72	96	400	1	3	17	32640 (Out of scan range)
720	32	144	32	540	10	7	11	39850
720	18	108	54	400	12	2	35	28320
720	28	48	102	400	1	4	45	28300
720	15	108	51	400	11	2	32	28322
1440	48	144	192	400	3	10	10	65500
1440	96	152	248	800	3	10	30	138500

[ash2fpga]

A couple questions:
Is this connected via analog i/o board vga or hdmi direct video to vga?
Do you have the ao486 OSD vsync setting set to 60hz or variable?

[Shogun]

Its connected hdmi to vga. Vsync settings doesn't make a difference. I also tried multiple converters and multiple monitors. Shaking is present in both settings. I'm just using the modeline that has the least shaking for now as I can't figure what in the modeline or settings could be adjusted to fix the issue.