Linux

xrayspx's picture

NoMachine NX Key Based Auth

Music: 

This will likely be updated.

I saw a request for some help in setting up key-based auth in NoMachine NX tonight just as I was going to bed and decided to do that instead.  I believe the request is for Mac-to-Mac, but for the moment I'm doing bi-directional Linux-to-Mac and Mac-to-Linux.  If I make any changes at all in how I set the Mac side up vs the Linux side I'll note them of course.

So here is the basic Linux client to Mac server.  In testing I set this up Linux-to-Linux.  The commands I used were exactly the same on the Linux and Mac servers.

Tags:
xrayspx's picture

Two Step Remote Assistance Tool

Music: 

My mom has a Mac, and occasionally something will fuck up in a way that is best fixed by me having some control over her machine.  I had one of those cases last week and it was embarrassing that there was no good way for me to get remote access.  Google Meet doesn't cut it, but there's a whole other Chrome Remote Desktop app, but that was a lot of hoops to install and gave up any hope of walking my mother through the install process.

xrayspx's picture

Mister Multisystem, Finally

Music: 

I am not writing this as a "getting started" or beginners guide, though it might evolve to serve that for some people. It's just notes I'm taking for things I need to look up. I'm not going to do any in-depth technical review or anything, go watch a youtube video for that. My perspective is from a career as a sysadmin and Linux user. Your mileage will certainly vary.

I strongly encourage you to skip my un-edited drivel and just scroll to more listy looking part.

This past weekend we resolved to kicking off, tidying up and playing with the MiSTer Multsystem. The actual "Problem" with getting this project going was that everything was way too easy to get, which has been our luck every time we need any Raspberry Pi's and stuff. We anticipated about a 3-4 month minimum lead time to land all the parts, figure out how it all goes together and how to get software to it. Everything turned up within a week. We got the DE-10 direct from the Terasic and they shipped pretty much immediately. Same story with the MultiSystem pack. I've been extremely interested watching this project as Neil has announced updates and features. The box was well packed and safe even after a real kicking by the various shipping companies.

I think it sat on the Project shelf for a month before we even had time around the holidays to put it together and try it out. It was super easy to assemble. Note: I spent 3 or 4 years in a local computer store in the '90s, and a 25-mumble-year career in IT so your mileage may vary. Actually that's not fair. Natalie did pretty much all the assembly work with me just kind of putting the case together. I'm still paranoid that I over-tightened screws or something since I've never really worked much with 3d printed anything. The case is very nice, well laid out and, while tight, everything is easy to route and very well documented.

Goals vs Reality


The Plan: This guy should sit prominently in The Room, quite probably wood-grained and hooked up to the main TV. There it would host all the console games for all the consoles neither of us ever had, and so have no muscle memory or nostalgia for. You know, For Kids.

The Reality: I was familiar from well-afar of the Mister project for some time. I know why the ST is there, and given all our other projects, this does have the capacity of seriously amusing me. But I wasn't sure about the MiSTer's interface, it looked like it might be a lot less friendly than something like EmulationStation. And that's a thing I go into below.

The other day I temporarily yet elegantly installed the MiSTer in the arcade cabinet and gave it a quick smoke test to make sure it still did things. I took a few minutes to figure out what using the external drive that I just happened to have hanging around was about and the effect it had on the virtual filesystem browser stuff.


1 week later and we spent a couple of hours today playing a bit, loading more games and software, then playing a bit more. Natalie wanted to start getting things documented for the Manual to the Living Room we are slowly beginning to put together. As we went through computer platforms from Atari 8-bit, ST, Amiga Natalie was taking notes on how to load software on each and get back to the main menu and whatever but really aside from necessary differences in how the keyboards are mapped, everything's just the same. We've been playing with an X-Box 360 controller because it's the only controller we own and it just worked. The arcade controls are going to take a bit of effort but the sticks and the trackball work, I just need to map buttons, and from what I can see that's not very hard.

The Big, Ugly Truth

The reason I really liked the standard linux machine plus a slick launcher is because it looks nice on the arcade cabinet and people could navigate it pretty easy. You know what? There are no "people" and this thing is way more convenient to use than the Pi + RetroPie. No trying to keep the button layout relatively synced between a bunch of different MAME emulators. Make that "A bunch of different versions of a bunch of different emulators". Nothing was ever consistent beyond the necessary controls. But it does look cool.

Enter the MiSTer. Since we're not emulating, there's not really anything much to tweak. Everything can pretty much be set up globally aside from a few platform specific changes. Even with arcade cores the UI is so consistent and easy to deal with that I won't exactly be pining for RetroArch any time soon. I need to figure out how to set a default set of controls though so I don't have to set the controls up for every single arcade cabinet forever. I know there's a menu item for that I just haven't pulled the trigger on possibly screwing things up yet.

It's also extremely easy to keep updated with all the latest cores using the "update_all" script. I'm using the "RetroDriven" fork, though I'm not sure what materially differs from the main line version. It's just the one I saw first. That tool was a revelation since you can just have it go out and populate any missing arcade ROMs, as well as adding cores that haven't made it to the main-line MiSTer distribution yet.

Sound can be /amazingly/ better than MAME. I'm not familiar enough with any console game to know what it should sound like vs an emulator. But I played a couple of games to test out Donkey Kong and it took me 3 games to get past just how much better it sounded on exactly the same hardware. I think it's like emulating a Moog synth vs copying it electrically at the circuit level. It'll at least get you a lot closer to "actual hardware" than MAME possibly can. (Important Note: The 4th game of DK I got within 700 points of my personal best 62,500 score on the board, and it's not like I'm constantly playing that game, it's pretty rare for me to do that well that quickly. [I think it's really more like 85k but that was before we had the board so I can't remember])

Conclusions?

While it's true that there are some arcade games that aren't yet "there" on the MiSTer, which work fine on a Pi-3, the fact is that while a given arcade core might not be ready for showtime just yet, like OutRun. And obviously there are systems that are simply too complex for the DE10-nano to copy. There are options, and they're imperfect, but still fun. The Sega 32x core works just fine I guess for games like Virtua Racing, and the Saturn core seems to be coming along fast. However for me a lot of the real value of this system is going to come in running the computers of my youth. Since I always had computers, I never really had consoles aside from the 2600 so I've been playing with the Atari 8-bit and ST cores (and the Amiga, because, you know, let's be real...). I'm even going to set up some hard drive images and profiles for color vs high-res mono.

This will get us through in emulation until the Next Big Kickstarter shows up at my door and I can use the ST to all of its ability with my SpecreGCR cartridge, etc...

I think we're into the MiSTer Multisystem for somewhere in the region of $400-$450. That's kind of steep, however what you get is every game released for every console up to about the PS1 / Sega 32x and maybe Saturn-ish. Plus all the 8-bit and 16-bit computers of the '70s through the early '90s, and their entire libraries. Plus a couple hundred arcade games. All easily managed in a simple to use and easy to understand interface.

Definitely 100% worth a look, especially given what people will spend for those "throwback" mini-consoles with a fixed set of titles baked into some potted-blob SoC.



I'm going to move a lot of the above into sorted bullets. I'm throwing some stuff around and will organize it as I go.



  • The "Minimig" Amiga emulator seems to run /way/ too fast, and I can't see any setting to slow it down, all indications are that it should be running a 7.whatever Mhz 68000. But the ST one works great. I saw someone saying the music was too fast, but indicated the game play was accurate. I disagree, the whole thing feels too fast to use for games I have muscle memory for (Tower Toppler).


  • As noted above in detail, the sound is fantastic, it's not even close.


  • Video Weirdness. I'm positive this is down to some scaling setting. I'm trying to get screenshots but my Mac keyboard doesn't have either a printscreen key or an F13 key, so balls. To use DK as an example since it does show up prominently what I'm seeing is variable "width" of identical single-pixel vertical lines or dots. So for instance the ladders in DK. One side of the ladder might be narrower than the other. Usually I see "normal" and "skinny" ladders, but I think I saw a "fat" one or two. Similarly in Ms. PacMan, some dots render as skinny. Aspect ratios seem correct otherwise so that's why I'm thinking it's some setting somewhere maybe. I don't even care it's a trade-off I'm willing to make for all the other benefits for gameplay and manageability.



    Yep, that was it: There are video processing options and you can choose from a list of profiles to match what you're doing


  • I do wish I could figure out how to directly edit core configs from the command line. Everything I've found seems to either be an empty file or binary that I can't modify. I'm sure there's something I can do. I'd love to figure out how to set generalized defaults for all cores, and then the user can define anything custom. So one key layout that will generally work across the board and then per-core settings can override that. Those per-core files could be shared as well. An editable "SNES Core - MS XBOX 360 Controller.cfg" or whatever.


  • I'm struggling to correctly make a folder for favorites using symlinks with absolute paths
  • xrayspx's picture

    Playlists

    Music: 

    Dr. Dre - Nuthin' But a G' Thang

    I had a request to share some playlist management stuff so I thought I should explain myself. I've got a significant CD collection, and a somewhat-significant collection of TV shows. This is fine on its own, but lots of media is pretty worthless without well curated playlists that you really don't have to think about. So I built Spotify, MTV and Syndicated TV.

    * NOTE: If you have a better way to do any of this let me know and I'll fix it. I particularly have the sense, which is not backed up by my testing, that "sort -R" isn't great.

    Music's easier so we'll start there. I use Strawberry to manage my music. This was all running under Clementine and aside from some DB schema changes, the scripts are portable between them.

    Until relatively recently I was never a big fan of "star" or "heart" ratings, but Clementine/Strawberry will store this metadata in the MP3 itself so I should be able to quickly recover if I lose my music database. In the app I have a few Smart Playlists like 3-Stars, 3 Stars + (This is 3, 4 and 5 star tracks), 4-Stars, 4-Star + and 5 Stars. To use 4 Star as an example, the rules look like this:

    Match every search term (AND)
    Rating - Greater than - 3.5 Stars
    Rathing - Less than - 5 Stars
    Ratin - Not Equals - 5 Stars
    Length - Greater Than - 8 Seconds

    That results in a playlist of 8423 songs with ratings between 4 and 4.99 stars. There was a bug in Clementine which I got fixed where ratings could exceed 5, so I'm a little careful to deal with weirdo cases, but it's pretty simple. I also have a bunch of manually selected playlists, so like an '80s one, '90s, and "Barn Radio". Barn Radio is our catch-all for the ubiquitous music we heard from the late '70s through late '80s. For Natalie that was largely with her dad in the dairy barn, for me it was the music of my 2 hours on the bus every day.

    Anyway, I have all these .m3us stored in a folder along with my MP3s called "playlists_base". These are used by a nightly playlist generator that pulls ~200 tracks and makes daily playlists running 8 or 10 hours each. The reason for this is that streaming software such as Airsonic-Advanced kind of chokes on massive playlists. It could be Airsonic itself, it could be populating the mobile client, I don't really know or care, other than to say it works great with list sizes under about 1000 tracks or so, so I keep them shorter.

    The x-Star playlists are all built from the database like this 4 Star + playlist below. You can see it do a couple of different Star Rating DB queries, dump out the tracks to $playlist_tmp.m3u, then cat that file and do a random sort to generate the final version. It's pretty easy to adjust the mix based on ratings, so if I wanted to weight high-rated tracks I could do that by adjusting how many tracks of the 200 are returned by each search:


    #!/bin/bash

    rm /Volumes/Filestore/CDs/playlists/4\ Stars\ +.m3u

    i=1

    while [ $i -le 100 ]
    do

    ### Switching from Clementine to Strawberry ###
    #       file=$(sqlite3 /var/tmp/clementine.db "select filename from songs where rating > "0.9" order by random() limit 1;" | awk -F "file://" '{print $2}')
            file=$(sqlite3 /var/tmp/strawberry.db "select url from songs where rating > "0.9" order by random() limit 1;" | awk -F "file://" '{print $2}')

            ### Clementine data encodes special characters and accent marks and stuff so I'm using
            ### Joel Parker Henderson's urldecode.sh to undo that: https://gist.github.com/cdown/1163649

            data=$(/home/xrayspx/bin/urldecode.sh "$file")
            if [ -f "$data" ]
            then
                    ### Have to escape leading brackets because grep treated it as a range and would allow duplicates ###
                    ### Can't do that in "data" because \[ isn't in the filename so they'll fail ###

                    escaped=$(echo "$data" | sed 's/\[/\\[/g')
                    #echo "$escaped"

                    ### Avoid duplicates
                    match=$(grep -i "$escaped" /var/tmp/4-star-tmp.m3u)
                    if [ -z "$match" ]
                    then
                            echo "$data" >> /var/tmp/4-star-tmp.m3u
                            ((i++))
                    fi
            fi
    done

    i=1

    while [ $i -le 100 ]
    do
    ### Switching from Clementine to Strawberry ###
    #        file=$(sqlite3 /var/tmp/clementine.db "select filename from songs where rating >= "0.8" and rating          file=$(sqlite3 /var/tmp/strawberry.db "select url from songs where rating >= "0.8" and rating 

            ### Clementine data encodes special characters and accent marks and stuff so I'm using
            ### Joel Parker Henderson's urldecode.sh to undo that: https://gist.github.com/cdown/1163649

            data=$(/home/xrayspx/bin/urldecode.sh "$file")
            if [ -f "$data" ]
            then
                    ### Have to escape leading brackets because grep treated it as a range and would allow duplicates ###
                    ### Can't do that in "data" because \[ isn't in the filename so they'll fail ###

                    escaped=$(echo "$data" | sed 's/\[/\\[/g')
                    #echo "$escaped"

                    ### Avoid duplicates
                    match=$(grep -i "$escaped" /var/tmp/4-star-tmp.m3u)
                    if [ -z "$match" ]
                    then
                            echo "$data" >> /var/tmp/4-star-tmp.m3u
                            ((i++))
                    fi
            fi
    done

    cat /var/tmp/4-star-tmp.m3u | sort -R > /Volumes/Filestore/CDs/playlists/4\ Stars\ +.m3u

    rm /var/tmp/4-star-tmp.m3u

    Those Star Rating lists are called at the beginning of my overall static playlist script, but the Barn playlist and other manually selected ones are built from the "playlists_base" directory. I basically just edit those .m3us in place with Strawberry as we add CDs. They just the files, do a random sort and pull the top 200. This will use any .m3u in .../playlists_base/ and make a daily file from it:


    #!/bin/bash

    #scp xrayspx@pro:~/.config/Clementine/clementine.db /var/tmp/

    ### Switching between Clementine and Strawberry ###
    #cp /Volumes/Filestore/CDs/playlists_base/clementine.db /var/tmp/

    cp /Volumes/Filestore/CDs/playlists_base/strawberry.db /var/tmp/

    /home/xrayspx/bin/3-star-playlist.sh
    /home/xrayspx/bin/4-star-playlist.sh
    /home/xrayspx/bin/5-star-playlist.sh
    /home/xrayspx/bin/get-the-led-out.sh

    ls /Volumes/Filestore/CDs/playlists_base/*.m3u > /Volumes/Filestore/CDs/playlists_base/m3us.txt

    while IFS= read -r file
    do

            filename=$(echo $file | awk -F "/Volumes/Filestore/CDs/playlists_base/" '{print $2}')

            echo Filename: $file

            rm "$file.full"
            rm "$file.scratch"
            rm "/Volumes/Filestore/CDs/playlists/$filename"

            ###Testing a change since Strawberry creates playlists without EXTINF lines ###
    #        array=`grep EXTINF "$file" | sort | uniq`
            array=`grep -v EXTINF "$file" | sort | uniq`

            printf '%s\n' "${array[@]}" | sort -R > "$file.full"
            head -n 200 "$file.full" > "/Volumes/Filestore/CDs/playlists_base/$filename.scratch"

            n=0
            while IFS= read -r extinfo
            do
    #       echo $extinfo
                    term=`echo $extinfo` # | cut -d "," -f 2-`
    #       echo $term

     ###Testing a change since Strawberry creates playlists without EXTINF lines ###
     # grep -A 1 -m 1 "$term" "$file" >> "/Volumes/Filestore/CDs/playlists/$filename"

            grep -m 1 "$term" "$file" >> "/Volumes/Filestore/CDs/playlists/$filename"
            done 

            rm "$file.full"
            rm "$file.scratch"

    done 

    rm /var/tmp/clementine.db
    rm /var/tmp/strawberry.db

    For TV shows it's a bit more complicated. I've got individual scripts for things like Sitcoms, Saturday Morning Cartoons, Buddy-Cop shows, Nick-at-Nite, etc. Each script uses a text file which just lists the relative path to the directories I want to randomize. I just read in that text file then scan each directory and build an array that again I sort -R and dump in an m3u. You'll see a couple of my conventions here, like the "dvd_extras" folders I use for any extras that I want to keep but don't want to have show up in the mix, as well as a bunch of other crap I grep out.

    This script references "./.sitcoms.txt", which looks like this:


    ./Archer (2009)
    ./30 Rock
    ./Absolutely Fabulous
    ./Alexei Sayle's Stuff


    #! /bin/bash

    array=$(
    while read line
    do
            find "$line" -type f;
    done < .sitcoms.txt
    )

    printf '%s\n' "${array[@]}" | sort -R | grep -v -w "batch" | grep -v dvd_extras | grep -v "./$" | grep -v "\.m3u" | grep -v -i ds_store |
     grep -v "\.nzb" | grep -v "\.nfo" | grep -v "\.sub" | grep -v "\.sfv" | grep -v "\.srt" | grep -v -i "\.ifo" | grep -v -i "\.idx" |
     sed 's/^/..\//' > ./1\ -\ Playlists/Sitcoms.m3u

    This dumps out to a folder called "1 - Playlists" inside my TV Shows directory, just so it shows up first. There's a folder in there for Blocks as well, in which I create blocks of 10 random episodes of a bunch of shows. This is built to replicate like TBS/TNT/USA in the evening where you just sit and watch a block of whatever is on. In practice I do this wrong and tend to be too picky about these and just watch blocks until I've worked my way through a whole series and wind up tired of it forever.

    One thing I do for things like Nick at Nite and overall Sitcom lists and stuff is that I mix in commercials. I don't do this very well though, I just treat my directory of commercials like any other TV show. I'd rather do "pull a TV show, toss in two commercials, repeat", but I'm not there yet I guess.

    The last type of lists I build are for music videos. I break this into a few different playlists, one overall catchall that pulls in all videos, a playlist for MTV 120 Minutes, and one for "Arcade / Pizzeria" music. Basically the ubiquitous music you'd hear in a pizza shop or arcade in the '80s or '90s. I do the same commercial thing here as well.

    Example:


    #! /bin/bash

    array=`find ../120\ Minutes -type f;
    find ../../../Commercials -type f`

    printf '%s\n' "${array[@]}" | sort -R | grep -v dvd_extras | grep -v "./$" | grep -v "ERRORS$" | grep -v "\.sh" | grep -v "\.m3u" |
     grep -v -i ds_store | grep -v ".nzb" | grep -v ".srt" > 120\ Minutes.m3u

    xrayspx's picture

    Kodi Machine Screensaver Notes

    Music: 

    Veruca Salt - Born Entertainer

    I've just spent too long messing with a small PC to replace my Raspberry Pi Kodi machine. Problem was that the system would blank the screen after 10 minutes and there's too much stuff to test, and each test takes 10 minutes. Make a change, reboot, wait 10 minutes, make another change, and so on.

    The problem was the Xorg default screen blanking, and it was fixed by creating /etc/X11/xorg.conf, with only the following config in it:

    Section "ServerFlags"
    Option "IgnoreABI" "True"
    Option "BlankTime" "0"
    Option "StandbyTime" "0"
    Option "SuspendTime" "0"
    Option "OffTime" "0"
    EndSection

    I had previously tried a bunch of stuff with setterm and enabling rc.local to run from Systemd, all to no avail, so I wanted to document this one for the next time.

    xrayspx's picture

    PiST

    Music: 

    Peter Murphy - The Sweetest Drop

    *Skip to the RetroPie customization stuff*

    Like every other moderately Vintage / Retro Computing person,
    not to mention my whole job being what it is, I immediately bought one of those
    12" IPS 16:9 Eyoyo monitors when I started seeing them pop up on some YouTube channels. Thing works great as a bench monitor, but it's kind of a weird size for use with 1980s OSes that expect 4:3. I found that Eyoyo also makes a 4:3 12" 800x600 monitor with all the same inputs so I grabbed one. Since the Atari branded monitors for the ST were 12", and I never saw any of these machines on anything much bigger than a 13" TV back in the day, this looks pretty much exactly as I remember and the correct aspect ratio makes everything feel "bigger" in the right way.

    The Pi 3 seems to be completely sufficient for emulating a stock 8Mhz 68000, it just needs enough power, swapping a 750ma power supply for 2.5A made a huge speed difference in emulation. I'm not trying to make this a "modern" experience like PiMiga or anything that requires any more horespower. For software, I started with RetroPie for their package management and the fact that they've already done the work of building all their packages to run from the command line with SDL/framebuffer, plus the ease of tweaking things like the boot splash screen, etc.

    Aside from that, it's really just Hatari and Amiberry. I found a 1GB ACSI disk image for the ST which is split into thirds and had a bunch of preloaded software. I've not added anything to it yet, but apparently the hero at 8bitchip has also archived over 1500 ST games and has patched them to run from a hard disk so I won't need to sort through a bunch of disk images which is great. I already spotted Oxyd in the list.

    That all makes me want to get NeoDesk running, which I gather is possible though it didn't immediately work in the 5 minutes I had to spend on it. Like I said, for this machine I'm not interested in running a "modern" take like PiMiga. But "nostalgia", well, while I'd certainly love to have original ST or Amiga hardware, I really am more nostalgic for the content and getting in front of Vroom or Nebulus for 10 minutes every 2 months, and it's awesome that it's using the same desktop I spent so much time in front of.

    The little boot menu I wrote just replaces the RetroPie autostart.sh file at /opt/retropie/configs/all/ with one that shows the user a menu to select the boot OS or shutdown. Windows 3.11 is a menu option, that's kind of TBD. DosBox is installed but I haven't done the full Windows install yet but it'll be fun to play Solitaire on this thing.


    Little tweaks I made to RetroPie

    I used RetroPie as my base rather than regular Raspbian because of their great packaging and basic customization tools. I just installed Hatari and Amiberry from the retropie_setup installer and they worked immediately at the command line. The retropie_setup tool also lets you swap out the default boot splash screen. I was expecting to just find the location of the file and swap in an Atari Fuji logo and be done with it. But they've built the tool into their setup utility, and even let you assign an MP4 so I was able to use an animated rainbow Fuji logo so it just looks awesome.

    I made one edit to /boot/cmdline.txt to suppress the bootup log output, so cmdline.txt looks like this now:

    console=serial0,115200 console=tty1 root=PARTUUID=8ee2ea28-02 rootfstype=ext4 fsck.repair=yes rootwait loglevel=3 quiet consoleblank=0 plymouth.enable=0

    I copied /opt/retropie/configs/all/autostart.sh out of the way and replaced it with:

    #!/bin/bash

    /home/pi/bin/menu

    That's pointing to the boot menu screen. It's really simple but really what am I trying to do here? I did add a countdown timer so it will automatically boot to an ST desktop after 20 seconds:


    #!/bin/bash
    # /opt/retropie/configs/all/autostart.sh

    clear
    echo "POMPEY PIRATES" | sed -e :a -e "s/^.\{1,$(tput cols)\}$/ & /;ta" | tr -d '\n' | head -c $(tput cols)
    echo ""
    echo ""
    echo "Press '1' For Atari ST"
    echo "Press '2' For Amiga"
    echo "Press '3' For Windows 3.11"
    echo "Press '9' For Bash Shell"
    echo "Press '0' For Shutdown"
    echo "Press 'T' For Trainer"
    echo ""
    echo ""

    msg="Booting Atari ST in"
    tput cup 12 0
    echo -n "$msg"

    l=${#msg}

    l=$(( l+1 ))

    for i in {15..01}
    do
    tput cup 12 $l
    echo -n "$i Seconds"

    read -t 1 -n 1 system 2>/dev/null

    if [ "$system" = "1" ]
    then
    /opt/retropie/emulators/hatari/bin/hatari --timer-d 0 && reset; /home/pi/bin/menu
    elif [ "$system" = "2" ]
    then
    /opt/retropie/emulators/amiberry/amiberry.sh && reset; /home/pi/bin/menu
    elif [ "$system" = "9" ]
    then
    exit 0
    elif [ "$system" = "0" ]
    then
    sudo shutdown -h now
    fi

    done

    /opt/retropie/emulators/hatari/bin/hatari --timer-d 0 && reset; /home/pi/bin/menu


    To-Do

  • I'll probably install Windows and Mini vMac just to do it
  • Add ST High Res mode
  • May add a timer to the menu to boot to the ST after some number of seconds just to make it more immersive.
  • Add a "POMPEY PIRATES" at the top and "PRESS 'T' FOR TRAINER" at the bottom for authenticity



  • xrayspx's picture

    Lattice of Convenience - MP3 Playlists

    Music: 

    Underworld - Kittens

    Hopefully everyone can live in the future someday.

    We do a lot with MP3 playlists. I run Airsonic for streaming around the house and in the car, and we have a playlist-based FM transmitter setup, etc. So I have scripts which run every night and generate playlists based on star ratings and other things (GET THE LED OUT ANYONE?).

    Previously what I've done is dump the contents of a bunch of Smart Playlists in Clementine to a file and use those files to generate the randomized 200 track daily playlists. The downside to that is that every time I add music or change star ratings, I'd have to refresh these "base" files like some kind of animal. I had base playlists for "3+ stars", "4+ Stars" and "5 Stars", among others.

    Today I decided to fix all that. Clementine uses a SQLite3 database, so now I'm just querying it instead, and it seems to be working well. For example, my "5 star" playlist in Clementine results in 10800 or so tracks. The same one built from the DB ends up with a couple hundred more tracks, but is pretty close. I'm not entirely sure what the difference there is just yet, but "close enough". What it looks like to me is I probably need to enable Samba case sensitivity.

    The DB records ratings as decimal numbers from 0.0 (Zero stars) through 1.0 (5 Stars). So to build a "4-Star +" playlist, searching for rating >= "0.8", you get ratings like this:

    1
    1.10000002384186
    0.800000011920929
    0.800000011920929
    0.800000011920929
    1.10000002384186
    1.10000002384186
    0.800000011920929
    0.800000011920929
    1
    1
    1
    1

    ! Caveat: Prior to Clementine 1.4.0rc1-533-gf4e70face there was a bug where it was possible to give a song a higher than 5 star rating (higher than 1.0 in the DB) as you can see above, so know that if you have Clementine from the repositories, it's likely you have that bug. For instance in the UI, if you want to show all 5 star songs, use "Rating is Greater Than 4.5 Stars" rather than "Rating is Equal to 5 Stars".

    Now I can just have a cron job to copy the master Clementine DB once a day to my server and drop it in next to the playlist generation scripts.

    The downside to all this is speed. When using the Clementine-Generated base playlists, I could be sure all the files actually exist on disk. However while Clementine will only show you files that exist in the UI, it doesn't seem to do a very good job of cleaning the database of stale files which no longer exist. So if you move or rename files, the old DB entries stick around unless you purge it completely and start over from scratch. That means I have to test every single file as I add it to the playlist, which takes time. It takes about 5-8 seconds to generate my 200 track 5-Star M3U file.

    The 5-Star.sh script is below if you'd like to play along at home:

      


    #!/bin/bash

    rm /Volumes/Filestore/CDs/playlists/5\ Stars.m3u

    i=1

    while [ $i -le 200 ]
    do
     file=$(sqlite3 ./clementine.db "select filename from songs where rating > "0.9" order by random() limit 1;" | awk -F "file://" '{print $2}')

     ### Clementine data encodes special characters and accent marks and stuff so I'm using
     ### Joel Parker Henderson's urldecode.sh to undo that: https://gist.github.com/cdown/1163649
     
     data=$(urldecode.sh "$file")
     if [ -f "$data" ]
     then
      ### Have to escape leading brackets because grep treated it as a range and would allow duplicates ###
      ### Can't do that in "data" because \[ isn't in the filename so they'll fail ###

      escaped=$(echo "$data" | sed 's/\[/\\[/g')
      #echo "$escaped"

      ### Avoid duplicates
      match=$(grep -i "$escaped" /Volumes/Filestore/CDs/playlists/5\ Stars.m3u)
      if [ -z "$match" ]
      then
       echo "$data" >> /Volumes/Filestore/CDs/playlists/5\ Stars.m3u
       ((i++))
      fi
     fi
    done

    For the 3+ and 4+ lists, I repeat this main block, but instead each rating dumps into a text file that I randomize into an .m3u at the end. So for the 3-Star + script below, I collect 130 5-star tracks, 45 4-star, and 25 3-star, push them out to a temp file and then cat temp.m3u | sort -R > "./3 Star +.m3u". I could do all this by creating a new table in the database and stuffing tracks into that, but this was faster for me to write and it works well enough:


    #!/bin/bash

    rm /Volumes/Filestore/CDs/playlists/3\ Stars\ +.m3u

    i=1

    while [ $i -le 130 ]
    do
     file=$(sqlite3 ./clementine.db "select filename from songs where rating > "0.9" order by random() limit 1;" | awk -F "file://" '{print $2}')

     ### Clementine data encodes special characters and accent marks and stuff so I'm using
     ### Joel Parker Henderson's urldecode.sh to undo that: https://gist.github.com/cdown/1163649
     
     data=$(urldecode.sh "$file")
     if [ -f "$data" ]
     then
      ### Have to escape leading brackets because grep treated it as a range and would allow duplicates ###
      ### Can't do that in "data" because \[ isn't in the filename so they'll fail ###

      escaped=$(echo "$data" | sed 's/\[/\\[/g')
      #echo "$escaped"

      ### Avoid duplicates
      match=$(grep -i "$escaped" ./3-star-tmp.m3u)
      if [ -z "$match" ]
      then
       echo "$data" >> ./3-star-tmp.m3u
       ((i++))
      fi
     fi
    done

    i=1

    while [ $i -le 45 ]
    do
      file=$(sqlite3 ./clementine.db "select filename from songs where rating >= "0.8" and rating

      ### Clementine data encodes special characters and accent marks and stuff so I'm using
      ### Joel Parker Henderson's urldecode.sh to undo that: https://gist.github.com/cdown/1163649

      data=$(urldecode.sh "$file")
      if [ -f "$data" ]
      then
       ### Have to escape leading brackets because grep treated it as a range and would allow duplicates ###
       ### Can't do that in "data" because \[ isn't in the filename so they'll fail ###

       escaped=$(echo "$data" | sed 's/\[/\\[/g')
       #echo "$escaped"

       ### Avoid duplicates
       match=$(grep -i "$escaped" ./3-star-tmp.m3u)
       if [ -z "$match" ]
       then
        echo "$data" >> ./3-star-tmp.m3u
        ((i++))
       fi
      fi
    done

    i=1

    while [ $i -le 25 ]
    do
      file=$(sqlite3 ./clementine.db "select filename from songs where rating >= "0.6" and rating

      ### Clementine data encodes special characters and accent marks and stuff so I'm using
      ### Joel Parker Henderson's urldecode.sh to undo that: https://gist.github.com/cdown/1163649

      data=$(urldecode.sh "$file")
      if [ -f "$data" ]
      then
       ### Have to escape leading brackets because grep treated it as a range and would allow duplicates ###
       ### Can't do that in "data" because \[ isn't in the filename so they'll fail ###

       escaped=$(echo "$data" | sed 's/\[/\\[/g')
       #echo "$escaped"

       ### Avoid duplicates
       match=$(grep -i "$escaped" ./3-star-tmp.m3u)
       if [ -z "$match" ]
       then
        echo "$data" >> ./3-star-tmp.m3u
        ((i++))
       fi
      fi
    done

    cat ./3-star-tmp.m3u | sort -R > /Volumes/Filestore/CDs/playlists/3\ Stars\ +.m3u

    rm ./3-star-tmp.m3u

    xrayspx's picture

    Caching Password passer

    Similar to the RDP Launcher, I occasionally need to grab passwords that I use all the time from KeePassXC to paste into various forms or prompts. Basically anything I use more than once per day, I have defined in this script for quick access. I don't want to be able to remember these, and I also don't want to have to interact with the password manager UI if I'm in a shell.

    xrayspx's picture

    DVD Ripping

    Music: 

    The Wipeouters - Ravin' Surf

    Another note for myself for later, and boy this is dumb.

    RDPLauncher

    TL;DR: Here's the Link:
    RDPLauncher

    I use RDP a lot and had some scripts to let me launch lots of RDP sessions without having to enter my random-generated passwords over and over. I wasn't happy with how I was handling those passwords so I've made it more secure using gpg and KeePassXC. Last night I made it compatible with Windows and MSTSC which will be uploaded here shortly once it's cleaned up a bit.

    Basically I'll click a shortcut for whatever host, which runs my launcher. I get prompted for my GPG passphrase, which reads from an encrypted file containing my KeePassXC passphrase, which is then used to retrieve the user password for launching the RDP session.

    Gpg-agent uses a cache-TTL to "hold the door open" for 10 minutes by default, so I can launch a bunch of sessions and only type my passphrase once.

    Requirements:

    - gpg client and running gpg-agent (gpg4win, etc) with a private key set up, etc.
    - cygwin if you're running Windows
    - KeePassXC (or some other key-store that has a command-line interface
    to query the database. In the beginning I was just using the gpg file
    with user/password pairs, so that works too)

    The tool has a few neat features:

    - If run from the command line with no arguments, it will prompt for user/pass/host/domain, good for one-off sessions to machines I won't log into much. That's great since I spend all my time in terminal windows and this stops me having to go back and forth to the mouse and keyboard while entering credentials.

    - If launched with -b, it prompts you for information for a one-off connection, but will also build a new shortcut launcher from a template. So like for the first connection to a machine you know you're going to use a lot. (Linux/Mac only)

    - Automatically tunnel sessions over ssh. This means I can launch RDP sessions on my Mac and they'll seamlessly proxy through my work laptop to the VPN.

    For tunneling, I am taking an arbitrary range of 200 ports and incrementing them based on what's currently listening. If there's already a process listening on port 6201, then try 6202 etc until there's an open one. So I can easily open 20-30 ssh tunneled sessions each with its own ssh process which will close down when the RDP window closes. 200 is "probably overkill", which means it might just be barely enough in the real world.

    The launcher shortcut mechanics are a bit different on my Linux and Mac machines so I split the -b script builder piece out based on OS. On Linux, I use KDE/Plasma, and so I generate these as KDE desktop files which look like this:

    #!/usr/bin/env xdg-open
    [Desktop Entry]
    Comment[en_US]=
    Comment=
    Exec=/home/xrayspx/bin/rdplauncher.sh -h it-host.xrayspx.com -d xdomainx -u xrayspx
    GenericName[en_US]=
    GenericName=host.xrayspx.com
    Icon=remmina
    MimeType=
    Name[en_US]=
    Name=host.xrayspx.com
    Path=
    StartupNotify=true
    Terminal=false
    TerminalOptions=
    Type=Application
    X-DBUS-ServiceName=host.xrayspx.com
    X-DBUS-StartupType=
    X-KDE-SubstituteUID=false
    X-KDE-Username=

    On the Mac side, I use shell scripts with the extension .rdp (which conflicts with Microsoft's client, but I don't care since I never use their client anyway). Those just launch using Terminal, so it does pop a terminal for a fraction of a second, but I really don't have a problem with that. To get the Terminal window to close (and I do associate these files with Terminal.app specifically rather than iTerm2), open Terminal.app, go to the Terminal menu -> Settings -> Profiles (tab) -> "Basic" or whatever profile is your default -> Shell (tab). Choose what action to take when the shell exits. I have it set to "Close if the shell exited cleanly" and "ask before closing" set to "only if there are processes other than the login shell..."

    The launcher for that looks like:

    #! /bin/bash
    rdplauncher.sh -h host.xrayspx.com -d xdomainx -u xrayspx &

    I generate those from the KDE .desktop files with a command like this:

    for host in $(ls | grep "\.desktop$" | awk -F ".desktop" '{print $1}'); do cmd=$(grep Exec $host.desktop | awk -F "xrayspx/bin/" '{print $2}'); echo "\!#/bin/zsh" >> $host.rdp; echo "$cmd &" >> $host.rdp; done

    That creates .rdp files in the same directory as the .desktop files, so now they can be moved around, have chmod set, etc.

    If I call it with AppleScript or Automator instead of a bash script as above, none of the password retrieval process works. I think it short circuits and sends the output back to the AppleScript rather than the bash script which ran the command. If I can get that working that would be ideal.

    The mechanics on Windows are similar to the Mac method. a .bat file which launches the bash script via Cygwin:

    C:\cygwin64\bin\mintty.exe -w hide -e /bin/bash -l -c '/home/user/bin/rdplauncher.sh -h host -u username -d domain'

    On Windows at least the Cygwin window it creates is hidden from the user, so that's nice.

    Pages

    Subscribe to RSS - Linux