Why not use a package manager?
Why not use a package manager?
Hello,
When i build my mister and start using it i found it strange to use all these script. I use linux for many years and package manager is really a core functionality, it's very practical and for me a big advantage over other OS. As Mister is base on linux i found it strange the project do not use some simple package manager for updates.
I mean right now i need to use 3x script to update my mister, official, llapi and jotego. Some of these script need to be edited to modify options (not very noob friendly but mister is diy so it's irrelevant). Also they are slow, the official will check all update including the one i don't want.
Also even if it's not completely useful for "classic" core, as you need to create a package for a simple rbf file, it could me more useless for arcade corde with MRA files. Create a package that include the core plus the right mra files, no need for the script to check if you have the right version of MRA. I guess it could also have some advantage to update other components like the linux side.
I know you will say "problem is you need to maintain a repo". Well github is (for me at least) not the best interface ever in that regards. There is a lot of package manager in the linux world, most are pretty mature and i suppose automation is strong now to build and push the packages to their repo.
When i build my mister and start using it i found it strange to use all these script. I use linux for many years and package manager is really a core functionality, it's very practical and for me a big advantage over other OS. As Mister is base on linux i found it strange the project do not use some simple package manager for updates.
I mean right now i need to use 3x script to update my mister, official, llapi and jotego. Some of these script need to be edited to modify options (not very noob friendly but mister is diy so it's irrelevant). Also they are slow, the official will check all update including the one i don't want.
Also even if it's not completely useful for "classic" core, as you need to create a package for a simple rbf file, it could me more useless for arcade corde with MRA files. Create a package that include the core plus the right mra files, no need for the script to check if you have the right version of MRA. I guess it could also have some advantage to update other components like the linux side.
I know you will say "problem is you need to maintain a repo". Well github is (for me at least) not the best interface ever in that regards. There is a lot of package manager in the linux world, most are pretty mature and i suppose automation is strong now to build and push the packages to their repo.
Re: Why not use a package manager?
Not the point, indeed solution exist and it's not bothering me that much to use scripts. I was just saying that this is strange to use scripts when there is a more reliable, easy and mature solution that is integrate in linux for very long.
-
- Top Contributor
- Posts: 1911
- Joined: Wed May 27, 2020 1:59 pm
- Has thanked: 145 times
- Been thanked: 454 times
Re: Why not use a package manager?
I agree with the update all script, it does it all plus some: Unofficial cores, Aracade zips. BIOS roms, Arcade Organizer. You can enable/disable features to your taste.
-
- Core Developer
- Posts: 547
- Joined: Sun May 24, 2020 9:30 pm
- Has thanked: 20 times
- Been thanked: 145 times
Re: Why not use a package manager?
Sounds like you're asking for somebody to do a bunch of work...
I think that people will perform the work that corresponds to a benefit to themselves.
If you were to create a solution that costs no ongoing support effort, it probably wouldn't be so hard to get people to join in, if they saw benefit (as in RBF + MRA)...
But to get somebody else to attack this problem, they would first need to share your view that it's a problem (and most currently don't). But even if they were to agree, they may choose a different way to solve the problem.
I think that people will perform the work that corresponds to a benefit to themselves.
If you were to create a solution that costs no ongoing support effort, it probably wouldn't be so hard to get people to join in, if they saw benefit (as in RBF + MRA)...
But to get somebody else to attack this problem, they would first need to share your view that it's a problem (and most currently don't). But even if they were to agree, they may choose a different way to solve the problem.
Re: Why not use a package manager?
I think what you're seeing can be summed up as MiSTer being set up (more or less) like an embedded Linux system rather than a traditional desktop or server distro. As far as the base Linux environment is concerned, the process is to update "the firmware" rather than updating individual packages. There are embedded Linux systems with package managers, but as far as I know it's unusual for them to be used for routine system maintenance. OpenWRT has opkg for example, but that's mostly used to install optional packages (since different routers have different amounts of storage), not for routine updates (since the preinstalled packages can't actually be removed from storage).
One complication for MiSTer is that most files that are routinely updated on MiSTer are typically on exFAT, which is not a traditional Unix file system and doesn't really support symlinks or permissions. I doubt that this is a supported and tested configuration for most Linux packaging systems (and I've personally had issues related to this in the past, e.g. trying to put /boot on FAT on a non-UEFI system).
I'm not saying you're wrong to want something based on a package manager; I've also sometimes wondered whether that might work better than the current system. I'm just saying that while MiSTer is very strange from a traditional distro viewpoint, it's only moderately strange from an embedded Linux viewpoint, and with its current architecture it has some quirks that don't fit very well into the typical desktop/server Linux model that traditional package managers were designed to support.
One complication for MiSTer is that most files that are routinely updated on MiSTer are typically on exFAT, which is not a traditional Unix file system and doesn't really support symlinks or permissions. I doubt that this is a supported and tested configuration for most Linux packaging systems (and I've personally had issues related to this in the past, e.g. trying to put /boot on FAT on a non-UEFI system).
I'm not saying you're wrong to want something based on a package manager; I've also sometimes wondered whether that might work better than the current system. I'm just saying that while MiSTer is very strange from a traditional distro viewpoint, it's only moderately strange from an embedded Linux viewpoint, and with its current architecture it has some quirks that don't fit very well into the typical desktop/server Linux model that traditional package managers were designed to support.
Re: Why not use a package manager?
Package managers have overhead and the MiSTer isn't a general purpose linux distribution. It's essentially an appliance. Once you go adding things like package managers you end up with things like optional packages and dependencies which complicate both maintenance and support.
Sure, if this was a general purpose platform it might make sense. But it's not. And if you're planning to turn it into something like that you're probably using the wrong device - a Pi running a general purpose distribution will be a better general purpose box.
They're cheaper, faster at the non-FPGA stuff, etc.
Sure, if this was a general purpose platform it might make sense. But it's not. And if you're planning to turn it into something like that you're probably using the wrong device - a Pi running a general purpose distribution will be a better general purpose box.
They're cheaper, faster at the non-FPGA stuff, etc.
-
- Top Contributor
- Posts: 622
- Joined: Fri Jan 22, 2021 4:36 pm
- Has thanked: 80 times
- Been thanked: 324 times
Re: Why not use a package manager?
Package management for frequently moving parts is a good idea in and of itself. It fixes all manner of edge cases of core management on a running MiSTer in a very proven way. As long as no permanently running daemon is required, the overhead of maybe a Python interpreter is not that big of a deal. You won't notice it's even sitting there when no code is executing.
I'm going ahead with a PoC for this and am currently examining different managers for suitability. Yum/DNF feels a bit heavy on dependencies, Apt seems leaner while functionally also more than adequate.
Pulling naked binaries from Github feels very baroque to me. The cobbled together update_all.sh even more so. I may bite myself in the butt in very grotesque ways with this but that's a good opportunity to learn.
Adding a package manager certainly does *not* mean that the whole OS needs to use it. I'm aiming for the most frequently moving parts first. FreeBSD has functioned like this for decades so there's good precedent.
I'm going ahead with a PoC for this and am currently examining different managers for suitability. Yum/DNF feels a bit heavy on dependencies, Apt seems leaner while functionally also more than adequate.
Pulling naked binaries from Github feels very baroque to me. The cobbled together update_all.sh even more so. I may bite myself in the butt in very grotesque ways with this but that's a good opportunity to learn.
Adding a package manager certainly does *not* mean that the whole OS needs to use it. I'm aiming for the most frequently moving parts first. FreeBSD has functioned like this for decades so there's good precedent.
- aberu
- Core Developer
- Posts: 1192
- Joined: Tue Jun 09, 2020 8:34 pm
- Location: Longmont, CO
- Has thanked: 247 times
- Been thanked: 411 times
- Contact:
Re: Why not use a package manager?
This isn't a desktop workstation or a server. It's a video game device that just happens to be running partially on linux. It's a very specific purpose use-case embedded device. Look up the philosophy behind JeOS (https://en.wikipedia.org/wiki/Just_enou ... ing_system) and you'll probably understand. It's just to play video games, not to be a Raspberry Pi replacement in it's entirety
birdybro~
- aberu
- Core Developer
- Posts: 1192
- Joined: Tue Jun 09, 2020 8:34 pm
- Location: Longmont, CO
- Has thanked: 247 times
- Been thanked: 411 times
- Contact:
Re: Why not use a package manager?
These kind of broad vague statements are not justified in and of themselves, you have to justify your position, so I'll respond to those attempts at justification below:
What edge cases does it fix? You didn't provide any examples at all relating to the MiSTer.It fixes all manner of edge cases of core management on a running MiSTer in a very proven way.
What if someone creates some absurdly complicated script that has problems that inadvertently affects the regular functionality of the user experience? Then the user doesn't know that this script is causing the problem, they come here asking for help. There is a real threat of people who help new users out being inundated with all kinds of requests for help despite the problem having nothing to do with the official MiSTer scripts and binaries. It already happens with the unofficial update_all script frequently, that script doesn't reboot after main+linux is updated, and proceeds to update, that's a problem and can cause corruption of the main mister binary. It also downloads unofficial cores which can cause users to be confused about them not working, since they aren't usually paying attention to what is official or unofficial.As long as no permanently running daemon is required, the overhead of maybe a Python interpreter is not that big of a deal. You won't notice it's even sitting there when no code is executing.
There are definitely scenarios like this for your equivalent proposed solution to your non-descript problems, that I'm sure you can imagine which would cause problems of their own.
Debugging based on user feedback can be hindered by unnecessarily added complexity, always an important thing to consider in systems design.
What benefit specifically is there to adding one of these package managers? Do you have any examples you could provide?I'm going ahead with a PoC for this and am currently examining different managers for suitability. Yum/DNF feels a bit heavy on dependencies, Apt seems leaner while functionally also more than adequate.
Just because something feels outdated doesn't mean it's not functioning, a hindrance, or in need of an update. Does it work? Is there a problem with it? etc...Pulling naked binaries from Github feels very baroque to me. The cobbled together update_all.sh even more so. I may bite myself in the butt in very grotesque ways with this but that's a good opportunity to learn.
BTW, this is awesome if you want to learn this! I am not trying to discourage you from that, but part of the learning in this way is to learn to ask and answer these kinds of questions too.
I agree that the package manager should be heavily restricted to certain use-case scenarios. I also think eventually the Menu core should only run as a different user without root privileges, a user named "MiSTer" perhaps. Locking this down may prevent people from making mistakes with unofficial scripts. Your package manager idea could fit into this security layout.Adding a package manager certainly does *not* mean that the whole OS needs to use it. I'm aiming for the most frequently moving parts first. FreeBSD has functioned like this for decades so there's good precedent.
Again, not trying to shit on the idea, just think it's important to be skeptical and have good reasons to add things to JeOS'
birdybro~
-
- Top Contributor
- Posts: 1321
- Joined: Thu Jun 11, 2020 2:31 am
- Has thanked: 15 times
- Been thanked: 213 times
Re: Why not use a package manager?
I wonder how all this is going to turn out the next time a full Linux partition image gets pushed and all of people's precious packages are wiped out =)
Re: Why not use a package manager?
@ dshadoff
I just discuss an idea, that's the point of open source, if the community doesn't like it let's move on. I don't ask anything to anyone.
@ ExCyber
I can see the point but in case of the router there is the firmware thing as you said. It's different on Mister, it act like a standard computer.
For the integration i don't see much where is the problem. Linux and it's file system have symlinks or permissions as you said. Maybe i'm wrong but these are feature and if you don't want to use them, well don't, it will just make the whole thing easier. Not sure any package manager cannot handle a more basic usage.
@ throAU
It's funny that you said it will add a lot of complexity when in fact it will remove it. Package doesn't need to be complicated with a lot of dependency, it's the app that require the complexity not the package, as it was said in linux there is a lot of thing to consider, dependency, symlink, permission etc... In Mister's case it will be simple version management, and basic package with rbf or rbf plus the set of MRA (and it would be even more logical so split the rbf and mra in two package). Very basic, only the linux / mister firmware part could probably be more tricky but i don't see how with my limited knowledge. The mister firmware is just a file that you need to update, remove the previous one and reboot.
And as Bas said it would be so much better than these binary push in github repo (with often it's very confusing interface) and dirty scripts for core / system update.
@rhester72
Don't understand what you mean
I just discuss an idea, that's the point of open source, if the community doesn't like it let's move on. I don't ask anything to anyone.
@ ExCyber
I can see the point but in case of the router there is the firmware thing as you said. It's different on Mister, it act like a standard computer.
For the integration i don't see much where is the problem. Linux and it's file system have symlinks or permissions as you said. Maybe i'm wrong but these are feature and if you don't want to use them, well don't, it will just make the whole thing easier. Not sure any package manager cannot handle a more basic usage.
@ throAU
It's funny that you said it will add a lot of complexity when in fact it will remove it. Package doesn't need to be complicated with a lot of dependency, it's the app that require the complexity not the package, as it was said in linux there is a lot of thing to consider, dependency, symlink, permission etc... In Mister's case it will be simple version management, and basic package with rbf or rbf plus the set of MRA (and it would be even more logical so split the rbf and mra in two package). Very basic, only the linux / mister firmware part could probably be more tricky but i don't see how with my limited knowledge. The mister firmware is just a file that you need to update, remove the previous one and reboot.
And as Bas said it would be so much better than these binary push in github repo (with often it's very confusing interface) and dirty scripts for core / system update.
@rhester72
Don't understand what you mean
Re: Why not use a package manager?
No, you are not 'reducing complexity'. you are moving it. Because now someone has to generate packages for what was previously just a dead simple release process for core developers. Now you're going to say 'automated CI/CD!' but now someone has to maintain that. And when it breaks (and it will) the core authors have almost zero visibility into it, and likely don't really have the expertise or motivation to debug it if they did. So all it does it delay their release or make it annoying to deal with.
I feel like you're trying to 'fix' MRAs by using packages and ignoring the fact that for everything else you're going to generate a package file for a single RBF, which seems a bit silly.
At the end of the day users are still going to run a script to update their mister, even if packages are involved. So now you have two problems.
What rhester72 means is that the 'linux' OS of the mister is updated as a single image, which is loopback mounted on boot. So when the 'linux firmware' of mister is updated, it completely replaces that image file.
I feel like you're trying to 'fix' MRAs by using packages and ignoring the fact that for everything else you're going to generate a package file for a single RBF, which seems a bit silly.
At the end of the day users are still going to run a script to update their mister, even if packages are involved. So now you have two problems.
What rhester72 means is that the 'linux' OS of the mister is updated as a single image, which is loopback mounted on boot. So when the 'linux firmware' of mister is updated, it completely replaces that image file.
-
- Core Developer
- Posts: 547
- Joined: Sun May 24, 2020 9:30 pm
- Has thanked: 20 times
- Been thanked: 145 times
Re: Why not use a package manager?
Yes, I had been wondering what the goal was here; what is intended to be 'fixed'.
I often hear people propose "solutions" without actually explaining the problems which they are intended to fix... the solutions are often presented as though they are the best/only solution available, for 'problems' which often aren't clearly explained (or perhaps even agreed upon).
From what I can tell from this thread, the "problem" being addressed by this is the issue of the MRAs sometimes failing to be updated ? (Anything else ?)
I often hear people propose "solutions" without actually explaining the problems which they are intended to fix... the solutions are often presented as though they are the best/only solution available, for 'problems' which often aren't clearly explained (or perhaps even agreed upon).
From what I can tell from this thread, the "problem" being addressed by this is the issue of the MRAs sometimes failing to be updated ? (Anything else ?)
- aberu
- Core Developer
- Posts: 1192
- Joined: Tue Jun 09, 2020 8:34 pm
- Location: Longmont, CO
- Has thanked: 247 times
- Been thanked: 411 times
- Contact:
Re: Why not use a package manager?
This is not a standard computer. It doesn't even act like one. There is no graphics processing unit. Basic functions of hardware often have to be routed through an FPGA core and basic connectivity that other standard computers have require significant development to even begin working.
It's funny that you said it will add a lot of complexity when in fact it will remove it. Package doesn't need to be complicated with a lot of dependency, it's the app that require the complexity not the package, as it was said in linux there is a lot of thing to consider, dependency, symlink, permission etc...
The way the MiSTer currently is setup the developers don't have to concern themselves with any of this, they just update their core on the github repository. Dependencies? For the RBF's? The only dependencies are those which are in the Main MiSTer binary, and linux image (for the most part). The RBF's are not the same as installing chromium on a raspberry pi. These are FPGA cores, they are binaries that describe to the FPGA how to configure it's hardware.
This is already done with the update script via the github api and naming schemes. No need to reinvent the wheel.In Mister's case it will be simple version management, and basic package with rbf or rbf plus the set of MRA (and it would be even more logical so split the rbf and mra in two package).
I think you should treat this, like Bas, as a learning opportunity. I'm glad you admit your limited knowledge. Update the package for some outdated software on a linux distro and tell me how it goes for you. Then try to add a package to a repository. Then come back and tell us how it's simpler than the scripts that are already in place that already do the job well.Very basic, only the linux / mister firmware part could probably be more tricky but i don't see how with my limited knowledge. The mister firmware is just a file that you need to update, remove the previous one and reboot.
Why is it better? Explain. I just see people saying it's better to move to a package manager without any justification. What's dirty about the scripts? How is the interface confusing? How could it be improved? etc...And as Bas said it would be so much better than these binary push in github repo (with often it's very confusing interface) and dirty scripts for core / system update.
birdybro~
-
- Posts: 2
- Joined: Thu Apr 01, 2021 5:17 am
- Been thanked: 2 times
Re: Why not use a package manager?
I'm a MiSTer newbie, but I do have 30-odd years experience wrangling Unix/Linux systems, and 15+ years managing people who do that job. I have to make sure thousands of individual machines are in perfect sync and interoperable with each other. One thing you learn really fast in that gig is that complexity is the enemy, and I have to remind the people I supervise not to over-think and over-engineer what they're doing.
Package managers are complicated, make messes, and are not the solution to all problems. Yes, they're useful in many use cases, but not all. Their job is to lay down a bunch of files that make up a particular package, and a package manager is a good way to pick and choose among the things you want on a particular machine, so you can install compilers or not, or a graphical desktop or not, or a CIFS server or not.
MiSTer isn't that multifaceted or complicated. Mine has 67 cores on it, and the whole MiSTer payload for cores, kernel, and whatnot looks to be under 1 gigabyte. That's astonishingly small for all the goodness that's packed into it. On a "normal" Linux server or desktop setup, you'll find individual packages larger than that, and there will be hundreds or thousands of packages installed.
The update_all.sh approach has the advantages of being very simple, with low overhead, and not having complicated package manager logic that can go off the rails and make poor decisions. In the MiSTer ecosphere, a core is a file, and to update you replace the file with the newer one. Done. The core is separate from the ROM and user data (disk images) and from what I see the update process doesn't mess with those. I think the worst case failure is that a bad core comes out and messes up somebody's gaming experience. The fix? The core gets fixed, goes into the repo, the user runs an update to pull down that one file, and life is good again.
Really I think a package manager is needless complication in this ecosphere.
Now the one thing I will object to in the existing update procedure is that it causes the root password to revert to the default and I had to find it in the wiki so I could log in and set it back. That took a few minutes of my life.
Package managers are complicated, make messes, and are not the solution to all problems. Yes, they're useful in many use cases, but not all. Their job is to lay down a bunch of files that make up a particular package, and a package manager is a good way to pick and choose among the things you want on a particular machine, so you can install compilers or not, or a graphical desktop or not, or a CIFS server or not.
MiSTer isn't that multifaceted or complicated. Mine has 67 cores on it, and the whole MiSTer payload for cores, kernel, and whatnot looks to be under 1 gigabyte. That's astonishingly small for all the goodness that's packed into it. On a "normal" Linux server or desktop setup, you'll find individual packages larger than that, and there will be hundreds or thousands of packages installed.
The update_all.sh approach has the advantages of being very simple, with low overhead, and not having complicated package manager logic that can go off the rails and make poor decisions. In the MiSTer ecosphere, a core is a file, and to update you replace the file with the newer one. Done. The core is separate from the ROM and user data (disk images) and from what I see the update process doesn't mess with those. I think the worst case failure is that a bad core comes out and messes up somebody's gaming experience. The fix? The core gets fixed, goes into the repo, the user runs an update to pull down that one file, and life is good again.
Really I think a package manager is needless complication in this ecosphere.
Now the one thing I will object to in the existing update procedure is that it causes the root password to revert to the default and I had to find it in the wiki so I could log in and set it back. That took a few minutes of my life.
-
- Top Contributor
- Posts: 622
- Joined: Fri Jan 22, 2021 4:36 pm
- Has thanked: 80 times
- Been thanked: 324 times
Re: Why not use a package manager?
As I said, I'm going to do this on my own system if nowhere else. When/if I get to a working state, all of you are welcome to use it or ignore it. I'm not going to write a massive essay here trying to convince anyone or seek permission.
So, I'll be back in this topic when I have something to show or to report my defeat.
So, I'll be back in this topic when I have something to show or to report my defeat.
Re: Why not use a package manager?
Not really, just update everything?
If everybody is running "update everything" and is synchronised to a known good combination of packages (the only one) you don't end up with edge cases where someone has version x of package y and it breaks. Failed downloads or installs half way through do not break the package manager because it doesn't exist.
I've been a linux admin since 1994 and have seen every package manager under linux (and FreeBSD for that matter, and home-brew under macOS) break in various ways that just doesn't need to be a thing.
I mean you do you.... I just don't think it is required for an "appliance" style device like MiSTer personally.
Re: Why not use a package manager?
Yes Mister act exactly like a standard computer, it's even simpler as you have access to the firmware directly on the sd card. In fact it was build like this to simplify it's use. Just browse your sd card or ssh, you can delete or update or whatever you want. It's not me who took the example of router, you can't access anything directly and if you want to update the firmware it's like a microcontroller or a lot of embedded solution, you need to flash it with specific tool, not something accessible to anyone. If Mister wasn't so simple i am sure it would have the success it has today, and even then there is still a few things that are not very noob friendly.aberu wrote: ↑Fri Apr 09, 2021 11:59 pmThis is not a standard computer. It doesn't even act like one. There is no graphics processing unit. Basic functions of hardware often have to be routed through an FPGA core and basic connectivity that other standard computers have require significant development to even begin working.It's funny that you said it will add a lot of complexity when in fact it will remove it. Package doesn't need to be complicated with a lot of dependency, it's the app that require the complexity not the package, as it was said in linux there is a lot of thing to consider, dependency, symlink, permission etc...
The way the MiSTer currently is setup the developers don't have to concern themselves with any of this, they just update their core on the github repository. Dependencies? For the RBF's? The only dependencies are those which are in the Main MiSTer binary, and linux image (for the most part). The RBF's are not the same as installing chromium on a raspberry pi. These are FPGA cores, they are binaries that describe to the FPGA how to configure it's hardware.This is already done with the update script via the github api and naming schemes. No need to reinvent the wheel.In Mister's case it will be simple version management, and basic package with rbf or rbf plus the set of MRA (and it would be even more logical so split the rbf and mra in two package).I think you should treat this, like Bas, as a learning opportunity. I'm glad you admit your limited knowledge. Update the package for some outdated software on a linux distro and tell me how it goes for you. Then try to add a package to a repository. Then come back and tell us how it's simpler than the scripts that are already in place that already do the job well.Very basic, only the linux / mister firmware part could probably be more tricky but i don't see how with my limited knowledge. The mister firmware is just a file that you need to update, remove the previous one and reboot.Why is it better? Explain. I just see people saying it's better to move to a package manager without any justification. What's dirty about the scripts? How is the interface confusing? How could it be improved? etc...And as Bas said it would be so much better than these binary push in github repo (with often it's very confusing interface) and dirty scripts for core / system update.
Yes i don't like the way mister handle it's update. You want specific about github, yes i can explain. Before i integrate my Mister into a case i spend some times doing things manually, i had no wired connection to my mister so i couldn't use scripts. Honestly what a mess github is, there is many files and folders, sometimes releases are moved from one repo to another, in sub directory (hello jotego), you can have outdated core in one folder and you need to move to the new one to get the new version. No big deals but it's not well organized as everyone do whatever they want, the interface is unpractical. You can clone the repo of course but you need to know what you do.
Now the scripts, well they are working indeed. But they are doing a lot of useless thing, so it's very slow, and you need to configure them. Mister is diy and it's obviously not very noob friendly in it's philosophy, but if the project want to simplify things, a simple and efficient package manager looks like a good choice to me.
I mean what is the more simple and user friendly, configure 3 scripts with your specific needs (so you need to take time to learn how the thing is working), then use the three of them separately or use an all in one script that apparently have problems. And when you use these scripts, even if you configure them to update only a few cores, theses things will take a lot of time checking updates for core you do not have and do not want, and keep checking for things you already have (like cheats etc). So you loose time for nothings.
In the other hand, you can have a simple package manager, a small app that you start, you choose the few package you wants, install them, then you can sometimes start the app again to update, the package manager will only update you mister base things (firmware and linux for exemple) and the cores that you had installed.
I don't know, but i keep thinking it would be more simple with a package manager.
What i don't understand is the assumption a package manager will add complexity and unreliability. Again i can understand in complex linux environment with all the package manager need to handle. I did have problems with dependency or strange behavior in my distro, but it's a linux distro, there is thousand of packages, with frequent big updates that change a lot of thing, with complicated dependency and rights... And even then problems were rare over the years, so in the Mister scenario when package manager will be use in a very simple way, i don't see where the dependency problem or weird decision by the package manager will be. And the "it is sure it will break someday", it's just a crystal ball thing.
What i understand is that no one want to take times to integrate it, and even less want to loose time maintain a repo even if some good automation could reduce this to a minimum. It is still extra works that could be done on core development i guess.
- aberu
- Core Developer
- Posts: 1192
- Joined: Tue Jun 09, 2020 8:34 pm
- Location: Longmont, CO
- Has thanked: 247 times
- Been thanked: 411 times
- Contact:
Re: Why not use a package manager?
Thanks for the reply with effort put into it, I appreciate it.
How does a package manager solve this problem if you aren't connected to the internet on your MiSTer?
1. Official updater script --> https://raw.githubusercontent.com/MiSTe ... /update.sh (no need to configure this)
2. Update-All script --> https://raw.githubusercontent.com/theyp ... ate_all.sh (I disabled the main mister updater, enabled jotego/bios_getter/mame+hbmame getter/arcade organizer scripts, disabled text scripts, I only have to configure this one time).
Both of these scripts have updater-pc versions which can be ran on your PC on the MicroSD removed from the MiSTer. So not sure what a package manager has to do with this.
I'm still not clear on the advantage that package management provides your specific scenario. I'm open to anything specific. If you mean you could run "apt update && apt upgrade jotego-cores", then this is already handled with the jotego updater script. If you mean you could run "apt update && apt upgrade jtcps1", then I guess you have a point. However, all of the update scripts could be modified to have command line arguments added to them in this same way (with variables so new cores could be included without extra work done by script makers each time there is a new core provided by the api), a package manager is not required for this feature to be implemented.
Either way, thanks for your thoughts, I found it interesting regardless.
If I understand you correctly you mean that you had to manually download and place things because you couldn't use scripts, is that correct? If that is the case, then just use updater-pc --> https://github.com/MiSTer-devel/Updater ... updater-pc
How does a package manager solve this problem if you aren't connected to the internet on your MiSTer?
Jotego chooses to do whatever he wants independently. Honestly you should take this up with him directly, he chooses to deviate from the MiSTer project's template --> https://github.com/MiSTer-devel/Template_MiSTer - and he's free to do that. Personally, using his updater script is not a problem for me, it works great. If you can't access the internet with the MiSTer, then use updater-pc from theypsilon. The problem you are referring to is already solved, albeit not immediately apparent to new users, admittedly.sometimes releases are moved from one repo to another, in sub directory (hello jotego)
This doesn't happen with the update scripts, they automatically remove the old cores. Yes if you do everything manually in this overly complicated way and choose not to use updater-pc, then you are going to have a tough time. You are making it hard on yourself in this case. A package manager will not solve this problem if you don't have internet access to the MiSTer in the first place.you can have outdated core in one folder and you need to move to the new one to get the new version.
I've only had to configure the update-all script. That's it. Nothing else. Here's the 2 scripts I run when I need updates:Now the scripts, well they are working indeed. But they are doing a lot of useless thing, so it's very slow, and you need to configure them.
1. Official updater script --> https://raw.githubusercontent.com/MiSTe ... /update.sh (no need to configure this)
2. Update-All script --> https://raw.githubusercontent.com/theyp ... ate_all.sh (I disabled the main mister updater, enabled jotego/bios_getter/mame+hbmame getter/arcade organizer scripts, disabled text scripts, I only have to configure this one time).
Both of these scripts have updater-pc versions which can be ran on your PC on the MicroSD removed from the MiSTer. So not sure what a package manager has to do with this.
The project prioritizes simplifying things for both developers and users as a balance, not just users. It's a breath of fresh air in the modern era where the push is solely for "end user" considerations, at the expense of everyone else. Maintaining packages entails more development than is currently required, and the scripts already handle every foreseeable advantage a package manager could provide for the developers.it's obviously not very noob friendly in it's philosophy, but if the project want to simplify things, a simple and efficient package manager looks like a good choice to me.
How do you know which package to update? Many users (myself included) do not actively pick and choose which specific core I want to update, we just run the script occasionally to see if there are updates. We don't follow all of the updates threads constantly, we don't follow the news, etc... If you run "apt update && apt full-upgrade -y" in Ubuntu 20.10 (full-upgrade is what is advised to use now instead of upgrade on newest versions btw, older versions use upgrade still), then it updates the package information (equivalent to running update.sh which downloads the latest mister-updater.sh), and the second command upgrades all of them by checking your local cache of the last ran updates against the api's response and downloads only the ones you need (skipping many checks). This is already basically doing package management but in a script. It's just more simple ("dumb") and doesn't check dependencies adaptively or anything like that. It gets the job done.In the other hand, you can have a simple package manager, a small app that you start, you choose the few package you wants, install them, then you can sometimes start the app again to update, the package manager will only update you mister base things (firmware and linux for exemple) and the cores that you had installed.
I'm still not clear on the advantage that package management provides your specific scenario. I'm open to anything specific. If you mean you could run "apt update && apt upgrade jotego-cores", then this is already handled with the jotego updater script. If you mean you could run "apt update && apt upgrade jtcps1", then I guess you have a point. However, all of the update scripts could be modified to have command line arguments added to them in this same way (with variables so new cores could be included without extra work done by script makers each time there is a new core provided by the api), a package manager is not required for this feature to be implemented.
I understand that feeling. The reason I am challenging your presuppositions is because you can learn from it. I love the MiSTer project because I learn something new everyday with it!I don't know, but i keep thinking it would be more simple with a package manager.
Complexity for developers. Developers currently just update their fork on github and submit a pull request, and the maintainer of the official mister repo in question decides whether to merge the pull request or not. Then as is necessary/valuable, the maintainer of the repo decides to update the binary (core_yyyymmdd.rbf). The script automatically grabs this. This already works. The complexity added would be in the form of having to update and test every single individual package for each of these cores. There isn't one script that does it all, there are individual packages for each and every core's binary now. You just multiplied the amount of relative "script" maintenance by every single core there is. Overall, this shouldn't be a problem if the same equivalent behavior as the current script regarding the github api calls is recreated in the package, but it's still extra stuff. Developers of these cores do this for fun, and they probably don't find package management to be very fun when compared to reverse engineering hardware.What i don't understand is the assumption a package manager will add complexity and unreliability.
I agree, this mindset that everything will break someday is not helpful. This applies to the current scripts too.And the "it is sure it will break someday", it's just a crystal ball thing.
Either way, thanks for your thoughts, I found it interesting regardless.
birdybro~
- kfreiberg
- Posts: 52
- Joined: Thu Sep 03, 2020 4:34 pm
- Location: Hoth (AKA Wisconsin)
- Has thanked: 7 times
- Been thanked: 7 times
Re: Why not use a package manager?
Hey Aberu,aberu wrote: ↑Sat Apr 10, 2021 5:08 pm I've only had to configure the update-all script. That's it. Nothing else. Here's the 2 scripts I run when I need updates:
1. Official updater script --> https://raw.githubusercontent.com/MiSTe ... /update.sh (no need to configure this)
2. Update-All script --> https://raw.githubusercontent.com/theyp ... ate_all.sh (I disabled the main mister updater,
Is there a benefit to running these both separately instead of just using the Update_All?
-
- Top Contributor
- Posts: 1911
- Joined: Wed May 27, 2020 1:59 pm
- Has thanked: 145 times
- Been thanked: 454 times
Re: Why not use a package manager?
I always use only the update_all script. Having them separate and disabling the main MISTer updater in the update_all script would have the advantage of shortening the update time: most of the time the main updater is enough to update the system. The rest concerns mainly the arcades but in my case I just run the update all script and let it do its thing, I am not in a hurry.
- aberu
- Core Developer
- Posts: 1192
- Joined: Tue Jun 09, 2020 8:34 pm
- Location: Longmont, CO
- Has thanked: 247 times
- Been thanked: 411 times
- Contact:
Re: Why not use a package manager?
Update_All doesn't reboot when Main and Linux updates, it just continues. My IT-spidey sense tells me that's generally not a good thing. I have seen my Main and Linux get corrupted when I have Update_All use the official updater, multiple times, and have correlated it mentally with Main and Linux getting updated at that time. Just something from personal experience.kfreiberg wrote: ↑Sat Apr 10, 2021 5:48 pmHey Aberu,aberu wrote: ↑Sat Apr 10, 2021 5:08 pm I've only had to configure the update-all script. That's it. Nothing else. Here's the 2 scripts I run when I need updates:
1. Official updater script --> https://raw.githubusercontent.com/MiSTe ... /update.sh (no need to configure this)
2. Update-All script --> https://raw.githubusercontent.com/theyp ... ate_all.sh (I disabled the main mister updater,
Is there a benefit to running these both separately instead of just using the Update_All?
Also thisjca wrote: ↑Sat Apr 10, 2021 5:57 pm I always use only the update_all script. Having them separate and disabling the main MISTer updater in the update_all script would have the advantage of shortening the update time: most of the time the main updater is enough to update the system. The rest concerns mainly the arcades but in my case I just run the update all script and let it do its thing, I am not in a hurry.
I coincidentally have the same mindset!
birdybro~
-
- Top Contributor
- Posts: 1911
- Joined: Wed May 27, 2020 1:59 pm
- Has thanked: 145 times
- Been thanked: 454 times
Re: Why not use a package manager?
You are right that update all does not reboot just after the update of Main and Linux, it does it only after running all the different scripts. So far I did not experience any problem and there has been quite a few reboots since I started to use the update all script.aberu wrote: ↑Sat Apr 10, 2021 6:05 pm ...
Update_All doesn't reboot when Main and Linux updates, it just continues. My IT-spidey sense tells me that's generally not a good thing. I have seen my Main and Linux get corrupted when I have Update_All use the official updater, multiple times, and have correlated it mentally with Main and Linux getting updated at that time. Just something from personal experience.
...
I take notice and will use the Main MISTer updater in this case but leave it in the update all script.
Re: Why not use a package manager?
This is quite easy.
If you have a MiSTer, and I guess you have, create a packaged based distro for it. You're free to do so! If it's any good, and works better than the current solution, and people like it, it will most probably be used instead of the current solution. IIRC that's how Mr Fusion came to be.
If you have a MiSTer, and I guess you have, create a packaged based distro for it. You're free to do so! If it's any good, and works better than the current solution, and people like it, it will most probably be used instead of the current solution. IIRC that's how Mr Fusion came to be.
-
- Top Contributor
- Posts: 622
- Joined: Fri Jan 22, 2021 4:36 pm
- Has thanked: 80 times
- Been thanked: 324 times
Re: Why not use a package manager?
There, @Solskogen said it. I'm not going to pop a new distro overnight but making quite decent progress with dpkg for the most dynamic bits already, after a few hours of experimenting, at zero operational impact to the current workflow for core developers.
-
- Posts: 49
- Joined: Wed Mar 24, 2021 12:52 pm
- Has thanked: 3 times
- Been thanked: 6 times
Re: Why not use a package manager?
If a package manager can be done for the mister, I do not see why that would be a bad thing. I use linux every day. It's easy to install, update, and remove packages with a single command or from a gui. I have made a package on aur repo. It's not hard.
Re: Why not use a package manager?
It is painfully obvious that some of you just want to play video games, and don't care about the linux underpinnings at all with regard to best practices. That is unfortunate.