This is not a problem, but an expected result for running FPGA cores with consumer TVs. The cores are developed by analysing the original hardware and try to stay as close to the original system timings as possible.
As home consoles were built with consumer TVs in mind, the analog output of the FPGA cores usually works fine with consumer TVs from the same region (PAL cores with PAL TVs (50Hz signal) and NTSC cores with NTSC TVs (60Hz signal) and many consumer TVs will display both 50Hz and 60Hz signals properly.
Arcade hardware was never intended to be used with consumer TVs, and correspondingly the cores will output varying analog signals. CRTs used in arcade cabinets can sync with a broader range of signals and can be adjusted to make the image fit the viewable area.
If you want to play different arcade cores on a consumer TV, your options are limited: Like mentioned in the comment above, you could try to adjust the image of your TV using the service menu. Usually this means to reduce the vertical and horizontal size to get less overscan in some cores, which will result in black borders in others. Some core authors offer different timings to help with compatibility, and you should test if they help in your specific case. Example: the R-Type core offers the original 55Hz and optional 50Hz and 60Hz timings. Please note that these options often change not only the output, but the timings of the core itself, resulting in slowdown or speedup.
You could also experiment with using custom video modes and the vga_scaler option in the MiSTer.ini configuration file. This basically fixes the output timings to create a stable picture (without overscan) on your TV and scales the core output to that video mode. It can be a time-consuming process and depending on the core resolution/modeline combination scaling artifacts can be visible. In conjunction with vsync_adjust=0 it even allows to run the display at a different refresh rate than the core output (which will also have visible artifacts like unsmooth scrolling). For finicky TVs, it is the only way to get any visible output at all with some of the arcade cores.
Please note that these limitations of analog output and CRTs are one of the reasons the MiSTer project exists. It comes with a powerful built-in scaler that can convert all these weird arcade signals to a high-resolution digital HDMI output. Scaling is highly customizable and comes with a huge selection of filters and masks to adjust the image to your liking.
