# Table of Contents [Search by distance](#Search-by-distance) [Method: WEIGHT](#Method-WEIGHT) [Method: DYNGENRE](#Method-DYNGENRE) [Method: GRAPH](#Method-GRAPH) [Filter: cultural](#Filter-cultural) [Filter: dynamic query](#Filter-dynamic-query) [Filter: influences](#Filter-influences) [Filter: similar artists](#Filter-similar-artists) [Tags & Weights: cultural](#Tags-amp-Weights-cultural) [Tags & Weights: related tracks](#Tags-amp-Weights-related-tracks) [Tags & Weights: similar artists](#Tags-amp-Weights-similar-artists) [Scoring methods](#Scoring-methods) [Scoring methods: chart](#Scoring-methods-chart) [Sorting: smart shuffle](#Sorting-smart-shuffle) [Sorting: harmonic mixing](#Sorting-harmonic-mixing) [Recipes & Themes](#Recipes-amp-Themes) [User descriptors](#User-descriptors) [Tagging requisites](#Tagging-requisites) [Tags sources](#Tags-sources) [Other tags notes](#Other-tags-notes) [Remove duplicates](#Remove-duplicates) [Scatter by tags](#Scatter-by-tags) [Global settings](#Global-settings) [Tags global remap](#Tags-global-remap) [Toolbar](#Toolbar) :::info This documentation may not be up to date. :mega: ::: # Search by distance Creates intelligent "spotify-like" playlists using high-level data from tracks similar to the currently selected one according to genre, style, key, etc. For ex, given a reference track: - Explore similar music from the same era - Follow the evolution of a genre during some period - Create a playlist transitioning from one genre to another seamleslly - Create an harmonic mix which sounds great no matter the genres mixed - Just a random playlist with some coherence in its order and tracks When their score is over 'scoreFilter', then they are included in the final pool. After all tracks have been evaluated and the final pool is complete, some of them are chosen to populate the playlist. You can choose whether this final selection is done according to score, randomly chosen, etc. All settings are configurable on the properties panel and/ or menus. These are the weight/tags pairs checked by default: | Weight | Tag | |-----------------|-----------------| | genreWeight | genre | | styleWeight | style | | dyngenreWeight | virtual tag | | moodWeight | mood | | keyWeight | key | | bpmWeight | bpm | | dateWeight | `$year(%date%)` | | composerWeight | composer | There are 2 custom tags which can be set by the user too: | Weight | Tag | |-----------------|--------------------| | customStrWeight | (empty by default) | | customNumWeight | (empty by default) | Any Weight/tags pair can be remapped and/or merged (sep. by comma). For example, linking genreWeight to 2 different genre tags on your files: | Tag | Remap | |-------|-------------------------------| | genre | `allmusic_genre,my_genre_tag` | All weight/tags pairs can be linked to TitleFormat Expr. Use tag names instead of TF expressions when possible (+ performance). For example, see `date`: TF is used to have the same results for tracks with YYYY-MM tags or YYYY tags. | Tag | Remap | |-----------|--------------------| | date | `$year(%date%)` | Genre and Style tags (or their remapped values) can be globally filtered. See `genreStyleFilter`. Case sensitive. For example, when comparing genre values from track A to track B, 'Soundtrack' and 'Radio Program' values are omitted: | Filter | Values | |---------------------|----------------------------| | genreStyleFilterTag | `Soundtrack,Radio Program` | There are 3 methods to calc similarity: WEIGHT, GRAPH and DYNGENRE. | Name | Method | Weights | |----------|--------------------------|---------------------------------------| | WEIGHT | Score filter | Uses genreWeight, styleWeight, moodWeight, keyWeight, dateWeight, dateRange, bpmWeight, bpmRange, composerWeight, customStrWeight, customNumWeight | | GRAPH | Score + Distance filters | Same than WEIGHT + max_graph_distance | | DYNGENRE | Score filter | Same than WEIGHT + dyngenreWeight | | Name | Description | |----------|-----------------------------------------------------------| | WEIGHT | Calculates similarity by score via tags comparison. Similarity is calculated by simple string matching (`Rock` $\neq$ `Soul`) and ranges for numeric tags. This means some coherence in tags is needed to make it work, and the script only works with high level data (tags) which should have been added to files previously using manual or automatic methods (like MusicBrainz Picard, see note at bottom). | | GRAPH | Apart from scoring, it compares the genre/styles tags set to the ones of the reference track using a graph and calculating their min. mean distance. For ex. for reference track with genre/styles (A,B,D), minimum mean distance is done considering (A,B,D) set against all library tracks. Lets consider a track with (A,B,C) genre/styles. First set matches at 2 points (0 distance), therefore only one path must be found, i.e. the shortest path from (A,B,D) to (C\) which will be $x$. Then the mean distance is calculated dividing the sum by the number of points of the reference track: $${0 + 0 + x} \over 3$$ Imagine Google maps for genre/styles, and looking for the distance from Rock to Jazz for ex. `max_graph_distance` sets the max distance allowed, so every track with genre/styles farther than that value will not be included in the final pool. Note this is totally different to simple string matching, `Acid Rock` may be similar to `Psychedelic Rock` even if they are totally different tag values (strings). This method is pretty computational intensive, we are drawing a map with all known genres/styles and calculating the shortest path between the reference track and all the tracks from the library (after some basic filtering). Somewhere between 2 and 5 secs for 40 K tracks. For a complete description of how it works check: 'helpers/music_graph_descriptors_xxx.js' And to see the map rendered in your browser like a map check: 'Draw Graph.html' | | DYNGENRE | Uses a simplification of the GRAPH method. Let's say we assign a number to every "big" cluster of points on the music graph, then we can simply put any genre/style point into any of those clusters and give them a value. So `Rock` would be linked to 3, the same than `Roots Rock` or `Rock & Roll`. It's a more complex method than WEIGHT, but less than GRAPH, which allows cross-linking between different genres breaking from string matching. For a complete description of how it works check: 'helpers/dyngenre_map_xxx.js' | Note about genre/styles ----------------------- GRAPH method doesn't care whether `Rock` is a genre or a style but the scoring part does! Both values are considered points without any distinction. Genre weight is related to genres, style weight is related to styles.... But there is a workaround, let's say you only use genre tags (and put all values together there). Then set style weight to zero. It will just check genre tags and the graph part will work the same anyway. Some tagging sources (Discogs) offer conglomerate of genre/styles as a single value. For ex. `Folk, World, & Country`. For all purposes, GENRE and STYLE tags are usually split into multiple values by comma, so it will be split into `Folk`, `World` and `& Country`. Obviously this will be unusable since the last value is not valid. In such case, tag may be remapped to something like `$replace(%GENRE%,', &',',')`. Tag properly your files, splitting the values into real genre/styles instead of conglomerates to avoid this situation or just don't use those values for GRAPH method, which makes no sense. Note about GRAPH/DYNGENRE exclusions ------------------------------------ Apart from the global filter (which applies to genre/style string matching for scoring purpose), there is another filtering done when mapping genres/styles to the graph or their associated static values. See `map_distance_exclusions` at 'helpers/music_graph_descriptors_xxx.js'. It includes those genre/style tags which are not related to an specific musical genre. For ex. "Acoustic" which could be applied to any genre. They are filtered because they have no representation on the graph, not being a real genre/style but a musical characteristic of any musical composition. Therefore, they are useful for similarity scoring purposes but not for the graph. That's why we don't use the global filter for them. This second filtering stage is not really needed, but it greatly speedups the calculations if you have tons of files with these tags! In other words, any tag not included in 'helpers/music_graph_descriptors_xxx.js' as part of the graph will be omitted for distance calcs, but you save time if you add it manually to the exclusions (otherwise the entire graph will be visited trying to find a match). Note about adding your own genre/styles to the graph ---------------------------------------------------- Instead of editing the main file, you can add any edit to an user configurable file which should be placed at the profile folder on: 'js_data/helpers/music_graph_descriptors_xxx_user.js'. See [search_by_distance_user_descriptors](#User-descriptors) readme for more info. Note about High Level data, tag coherence, automatic tagging and Picard ----------------------------------------------------------------------- Network players and servers like Spotify, Itunes Genius, YouTube, etc. offer similar services to these scripts. Whenever you play a track within their players, similar tracks are offered on a regular basis. All the work is done on the servers, so it seems to be magic for the user. There are some important caveats for this approach: It only works because the tracks have been previously analyzed ('tagged') on their server. And all the data is closed source and network dependent. i.e. you can always listen to Spotify and your great playlist, at least while you pay them, those tracks are not removed for your region and you have a constant Internet connection. That music listening model has clear drawbacks, and while the purpose of this caveat is not talking about them, at least we have to note the closed source nature of that analysis data. Spotify's data is not the same than Youtube's data... and for sure, you can not make use of that data for your library in any way. Ironically, being at a point where machine learning and other methods of analysis are ubiquitous, they are mostly relegated behind a pay-wall. And every time a company or a developer wants to create similar features, they must start from scratch and create their own data models. An offline similar program which does the same would be [MusicIP](https://spicefly.com/article.php?page=what-is-musicip). It appeared as a viable alternative to that model, offering both an Internet server and a complete ability to analyze files offline as fall-back. Nowadays, the company is gone, the software is obsolete (although usable!) and documentation is missing for advanced features. The main problems? It was meant as a standalone solution, so there is no easy way to connect other programs to its local server to create playlists on demand. It can be done, but requires manually refreshing and maintaining the server database with new tag changes, data analysis, and translating ratings (from foobar for ex.) to the program. The other big problem is analysis time. It may well take a minute per track when calculating all the data needed... and the data is also closed source (so it has no meaning outside the program). The reason it takes so much time is simple, the program was created when machine learning was not a reality. MusicIP may well have been ahead of its time. Back to topic, both online and offline methods due to its closed source nature greatly difficult interoperability between different programs and use-cases. These scripts offer a solution to both problems, relying only in offline data (your tags) and open source data (your tags again). But to make it work, the data (your tags) need relevant info. Since every user fills their tags without following an universal convention (most times leaving some tags unfilled), the only requirement for these scripts to work is tag coherence: - Tags must point to high-level data, so analysis (whether computational or human) must be done previously. 'Acoustic' or 'Happy' moods are high level data, 'barkbands' with a list of values is low-level data (meant to be used to calculate high-level ones). For more info: https://acousticbrainz.org/data - Tags not present are simply skipped. The script doesn't expect any specific tag to work, except the obvious ones (can not use GRAPH method without genre/style tags). This is done to not enforce an specific set of tags, only the ones you need / use. - Tags are not hard-linked to an specific tag-name convention. Genre may have as tag name `genre`, `my_genre` or `hjk`. - Basic filtering features to solve some corner cases (for genre/styles). - Casing must be the same along all tags. i.e. `Rock` always spelled as `Rock`. Yes, it could be omitted using .toLowerCase(), but is a design decision, since it forces users to check the casing of their entire set of tags (ensuring a bit more consistency in other fields). - Reproducibility all along the library set. And this is the main point of tag coherence. About Reproducibility: If two users tag a track with different genres or moods, then the results will be totally different. But that's not a problem as long as every user applies its own 'logic' to their entire library. i.e. if you have half of your library tagged right and the other half with missing tags, some wrongly set and others 'as is' when you got the files... then there is no coherence at all in your set of tracks nor your tags. Some follow a convention and others follow another convention. To help with that here are 2 advises: tag your tracks properly (I don't care about specific use-cases to solve what ifs) and take a look at MusicBrainz Picard (that's the open source part): https://picard.musicbrainz.org/ Now, you DON'T need to change all your tags or your entire library. But Picard offers 3 tags using the high-level open source data of AcousticBrainz: mood, key and BPM. That means, you can use your manual set tags along those automatically set Picard's tags to fulfill both: your tag convention and reproducibility along your entire library. Also you can manually fix or change later any mood with your preferred editor or player. Picard also offers plugins to fill other tags like genres, composers, etc. Filling only empty tags, or adding them to the existing ones, replacing them, ... whatever you want. There are probably other solutions like fetching data from AllMusic (moods), lastFm (genres), Discogs (composers), etc. Use whatever you want as long as tag coherence and reproducibility are ensured. What happens if you don't want to (re)tag your files with moods, bpm, key, splitting genres/styles, ...? Then you will miss that part, that's all. But it works. What about a library properly tagged but using `rock` instead of `Rock` or `African Folk` instead of `Nubian Folk` (although that's a bit racist you know ;)? Then use substitutions at 'helpers/music_graph_descriptors_xxx.js' (not touching your files) and/or use mass tagging utilities to replace values (touching them). And what about having some tracks of my library properly tagged and not others? Then... garbage in, garbage out. Your mileage may vary. # Method: WEIGHT Creates intelligent "spotify-like" playlists using high-level data from tracks similar to the currently selected one according to genre, style, key, etc. When their score is over 'scoreFilter', then they are included in the final pool. WEIGHT presets work by matching genre\style tags (Rock $\neq$ Hard Rock), without using complex logics. Therefore they work with arbitrary tag values. See [main readme](#Search-by-distance) (ALL) for more usage info. # Method: DYNGENRE Creates intelligent "spotify-like" playlists using high-level data from tracks similar to the currently selected one according to genre, style, key, etc. When their score is over 'scoreFilter', then they are included in the final pool. DYNGENRE presets work by grouping similar genre\style tags (Rock group: Rock, Hard Rock, Alt. Rock, ...). Instead of using tag matching (Rock $\neq$ Hard Rock) or more complex linking. Tags must be included on '.\helpers\dyngenre_map_xxx.js' to work. See [main readme](#Search-by-distance) (ALL) for more usage info. # Method: GRAPH Creates intelligent "spotify-like" playlists using high-level data from tracks similar to the currently selected one according to genre, style, key, etc. When their score is over 'scoreFilter', then they are included in the final pool. See '_images\search_by_distance_GRAPH_diagram.png'. GRAPH presets work by linking similar genre\style tags on a graph (Rock → Southern Rock → Country Rock → Country). Instead of using tag matching (Rock $\neq$ Hard Rock) or simple grouping (Rock group: Rock, Hard Rock, Alt. Rock, ...). Tags must be included on '.\helpers\music_graph_descriptors_xxx.js' to work. See [main readme](#Search-by-distance) (ALL) for more usage info. # Filter: cultural Cultural similarity between artists and/or genres (or styles) maybe be taken into account via query filters, which are applied before scoring and comparing tracks. Source ------ Country tags (aka LOCALE) are retrieved from 'LOCALE LAST\.FM' file tag or [World-Map-SMP database](https://github.com/regorxxx/World-Map-SMP). Filters: (pre-scoring filters) ------------------------------ - Artist cultural filter: by same/different continent, region or country than the selected track's artist. - Genre cultural filter: by same/different continent or region than the selected track's genre/styles. Note a genre may be associated to multiple regions, contrary to artists. # Filter: dynamic query Filters the library by a query built according to the reference track or theme. For filtering purposes ---------------------- - Empty queries and 'ALL' are non valid (they do nothing). - Don't add 2 tags which will be remapped internally by Search by Distance tags (see below) in the same query. Use the multiple slots for that. - When used along themes, since there is no reference track, the theme tags are used instead. Note this will probably limit the usage to internal remapped tags, since other tags like 'ARTIST' are obviously missing on the theme. In such case that query is omitted. The tag may be manually added to the theme though, by editing the theme json. Rules ----- - Tags are remapped with the current settings for tags. i.e. 'GENRE' could be remapped to 'GENRE' and 'ALLMUSIC GENRE' before execution, if the current genre tag is remapped that way. This is by design, and always applied before further processing. - Dynamic variables are allowed too, enclosed on '#': `#ARTIST#`, `#$year(%date%)#`, ... - There are a few special variables, which are not evaluated against a track but globally. For ex: `#YEAR#` → 2023 - These placeholders will be replaced with focused item\'s TF result on execution. - Using tags alone, translates into multivalue tags/queries by default. - When using TF expressions (`$...`), tags are not converted and the expression is executed 'as is'. - When using a NOT along a remapped tag, the evaluation will be joined with an `AND` instead of `OR`. - Standard queries may be added -ONLY- to the end of a dynamic query expression. In any other case it will probably throw an error. - Multiple dynamic multivalued variables are allowed, but they must be enclosed on parenthesis to work. - In any case queries are tested before using it, so errors will be shown as a popup. Special variables ----------------- Don't require a track to be evaluated (i.e. they always work with themes): - `#DAY#` → 14 (current day) - `#MONTH#` → 11 (current month) - `#YEAR#` → 2023 (current year) Some examples ------------- For a rock track: `GENRE IS #GENRE#` → `GENRE IS Rock`. The entire library would be filtered with tracks from that genre. If genre tag is remapped (with 2 tags), then it would translate into: `GENRE IS #GENRE#` → `(GENRE IS Rock) OR (ALLMUSIC GENRE IS Rock)` For a rock and electronic track, it would translate into: `GENRE IS #GENRE#` → `(GENRE IS Rock) AND (GENRE IS Electronic)` Note multi-value tags are split and produce multiple query entries by default. (*) If genre tag is also remapped, in such case it would look like: `GENRE IS #GENRE#` → `((GENRE IS Rock) AND (GENRE IS Electronic)) OR ((ALLMUSIC GENRE IS Rock) AND (ALLMUSIC GENRE IS Electronic))` It can also be used to remap tags like this (for a Bob Dylan's track): `COMPOSER IS #ARTIST#` → `COMPOSER IS Bob Dylan` Merging standard and dynamic expressions is also possible, as long as the standard query expression is added at the end, like: `GENRE IS #GENRE# AND NOT (%RATING% EQUAL 2 OR %RATING% EQUAL 1)` → `(GENRE IS Psychedelic Rock) AND (GENRE IS Hard Rock) AND NOT (%RATING% EQUAL 2 OR %RATING% EQUAL 1)`. Multiple dynamic queries are also possible. Note how the parenthesis are used to limit dynamic expansion. `((GENRE IS #GENRE#) OR (STYLE IS #STYLE#))` → `(((GENRE IS Psychedelic Rock) AND (GENRE IS Hard Rock)) OR ((STYLE IS Acid Rock) AND (STYLE IS Live)))` On other cases parenthesis may be redundant (but valid): `((DATE IS #DATE#) OR (DATE IS #$add(%DATE%,1)#))` → `((DATE IS 1969) OR (DATE IS 1970))`. Note how single valued tags don't need extra parenthesis since dynamic expansion does not apply here: `(DATE IS #DATE# OR DATE IS #$add(%DATE%,1)#)` → `(DATE IS 1969 OR DATE IS 1970)` Don't forget the rules for internal tag remapping about `NOT`: `GENRE IS #GENRE#` → `GENRE IS Rock` `NOT GENRE IS #GENRE#` → `NOT GENRE IS Rock` With internal tag remapping, the second case uses `AND` instead of `OR`: `GENRE IS #GENRE#` → `(GENRE IS Rock) OR (MYGENRE IS Rock)` `NOT GENRE IS #GENRE#` → `(NOT GENRE IS Rock) AND (NOT MYGENRE IS Rock)` Common errors ------------- Queries don't allow TF expressions after IS/HAS/GREATER/... terms. For ex: `"$sub(%DATE%,10)" GREATER #DATE#` → `"$sub(%DATE%,10)" GREATER 1969` And `%DATE% GREATER #$add(%DATE%,10)#` → `%DATE% GREATER 1979` Are equivalent and valid, since both ensure tracks with date > 1979. But this similar query is non valid (and will throw an error): `%DATE% GREATER "$add(#DATE#,10)"` → `%DATE% GREATER "$add(1969,10)"` Since the TF expression is not executed during query evaluation! Note in these cases there is (usually) no need to add quotes at the dynamic term when using a function since the final value must be a string/number, not another function (like the last example). It is not possible to add 2 tags which are going to be remapped at the same query slot. For ex: `NOT GENRE IS #GENRE# AND STYLE IS #STYLE` (*) Since `#TAG# IS VALUE` forces tag to be equal to value, muti-value tags follow the same rationale. If you want to use another logic (OR, combinations, etc.) between different values of the same tag then the tool to use would be 'Search Same By'. # Filter: influences Filters the library by a query built according to the genre/styles of the reference track or theme. This query is built on-the-fly taking into consideration which other genres/styles are influences or anti-influences according to the genre/style graph. Note distance scoring usually filter anti-influences by design, since they are further away than other genre/styles. But applying a query ensures there are zero tracks with those, no matter the other settings... or that only influences are retrieved (rock tracks would never output rock tracks -which are naturally similar- but their influences!). Rules ----- - Only works on 'GRAPH' mode. Filters: (pre-scoring filters) ------------------------------ - Anti-influences filter: removes any track considered to be an anti-influence to the genre/styles of the reference track (or theme). - Conditional anti-influences filter: applies the previous filter for specific genre/styles only (found at 'style_anti_influences_conditional' on the descriptors). For any other genre, the filter is not applied. For ex. a Jazz track as reference will use it, but a Rock track will skip it. Some genres are more sensitive to style changes than others; it may be tolerable to get a grunge track on a rock playlist, but that situation should be absolutely avoided for Jazz playlists. - Influences filter: only uses tracks considered to be an influence/derivative to the genre/styles of the reference track (or theme). Note there is no distinction between influences (past) and derivatives (future), that's taken into account with the 'DATE' scoring and/or other query filters. For ex., to get influences from the past, use a dynamic query to only retrieve influences with a date lower than the current one. There are some filters specific for this by default on the settings. # Filter: similar artists Filters the library by a query built according to the similar artists of the reference track. This query is built on-the-fly using the file tags or JSON database (in both cases they must be computed first). Note it will always prefer file tags, if present, over any JSON database; both are never merged. This is done to ensure a possibility to override the database with a preferred value (tags) if desired. If that's not desired, just don't use similar artist tags or update them on demand. # Tags & Weights: cultural Cultural similarity between artists and/or genres (or styles) maybe be added via query filters or special tags. Source ------ Country tags (aka LOCALE) are retrieved from 'LOCALE LAST\.FM' file tag or [World-Map-SMP database](https://github.com/regorxxx/World-Map-SMP). Special Tags: (tags and weighting) ---------------------------------- - Artist Region: uses the selected track's artist as reference, and compares it to every artist's country on library according to an allowed range. Distance is calculated by same country (0), same region (1), same continent (2) or different continent(4). - Genre/Style Region: uses the selected track's genre/styles as reference, and compares it to every other track on library according to an allowed range. Distance is calculated by same genre (0) or another genre from same region (1), same continent (2) or different continent(4). Note a range of 4 would make values found at a different continent to give a score of zero for the given tag, so by default the ranges are set to 5. In such case the score (in linear mode) gives a 20%. - Same value: $(5 - 0) / 5 = 100\%$ - Same region: $(5 - 1) / 5 = 80\%$ - Same continent: $(5 - 2) / 5 = 60\%$ - Diff. continent: $(5 - 4) / 5 = 20\%$ A lower range would completely discard farthest values in the map. Negative scoring may also be enabled. For ex. range equal to 3: - Same value: $(3 - 0) / 3 = 100\%$ - Same region: $(3 - 1) / 3 = 66\%$ - Same continent: $(3 - 2) / 3 = 33\%$ - Diff. continent: $(3 - 4) / 3 = -33\%$ (or $0\%$) (*) Note there is always a bias towards values within the same continent. (*) May be negative if the related setting for values out of range is used. # Tags & Weights: related tracks Tracks considered too similar/different by the user are handled with special tags which identify these relations. Source ------ Tags are retrieved by TF (can be remapped), by default: - `RELATED` - `UNRELATED` These tags can contain any mix of any of these values: - TITLE - ARTIST - MUSICBRAINZ_TRACKID: is an unique ID associated to a recording, so there is no possibility of collisions with tracks with same title or artist but not being the same recording. Special Tags: (tags and weighting) ---------------------------------- - Related: if the reference track has any value on this tag, then any other track matching by TITLE, ARTIST or MUSICBRAINZ_TRACKID such values will be considered more similar. The weight is configurable. In any case the max score will be 100. - Unrelated: if the reference track has any value on this tag, then any other track matching by TITLE, ARTIST or MUSICBRAINZ_TRACKID such values will be considered less similar. The weight is configurable. In any case the min score will be zero. - This additional score is added/subtracted to the main score calculations, so it's only applied in case the reference track has such tags. OTherwise it's simply skipped. For this reason, there is no 'Base score' or 'Scoring method' settings available, since the weight is added as is. For ex. a related track just gets +25 score points if matched. - Setting related tracks weight to 100 will ensure they are always shown on the output playlist. - Setting unrelated tracks weight to -100 will ensure they will never be shown on the output playlist. # Tags & Weights: similar artists Search by Distance tools may be used to compute the similar artists to those from the currently selected tracks (duplicates are filtered first). This option may be found on the customizable button and Playlist Tools. It tries to compute something equivalent to 'Similar Artists Last\.fm', which can be retrieved with Bio script (but requires internet). In this case the process is entirely offline and uses the already coded routines of Search by Distance to find which artists are usually the most similar to a random set of [N=50] tracks by every selected artist. It requires some time, since it's equivalent to perform the regular 'GRAPH' search x N times x selected artist(s). The advantage is that it does not requires internet and works with any artists, whether it's 'popular' or not, since it's not based on things like users' listening behaviors, eurocentric bias, etc. Results are also limited to what the user has on library, instead of being based on an external list (static). The results are output to console and saved to JSON, so they only need to be calculated once. Later changes to the graph descriptors may affect the results, in that case the same artists may be checked again to update the list if required. Search by Distance offers multiple filtering options based on those results, which may be used on the customizable button or via recipes. Script uses both the JSON file and tags (take precedence if present). After scanning, results (from JSON) may also be saved to tags for all the tracks present on library by the analyzed artists. On tags, other tools like Playlist Tools may use them to easily create queries, pools, etc. Tag is named 'SIMILAR ARTISTS SEARCHBYDISTANCE' (but it may be changed at '[FOOBAR_PROFILE]\js_data\presets\global\globTags.json'). NOTE: Estimated processing time will be shown on popups, have that in mind when processing a large library (it may be better to process it on different steps). It doesn't mind how many tracks you select, only how many different artists are found within that selection! Calculation time is proportional to the number of tracks on library squared though. Also note results are heavily dependent on tags, so if you totally change the tags from an artist discography, it would be recommended to rerun the analysis... obviously, adding new albums/tracks from an already properly tagged artist does not count here, since adding more 'Rock' tracks by Aerosmith to your library will not change anything if all the existing tracks by Aerosmith were already representative of their main work (Rock). In any case think of your library like an Spotify server: whenever you add/edit tracks there, the global statistics change and also the way tracks and artists inter-relate. It's a live thing. Use your own criterion about rerunning analysis when needed, considering the time/exactness equation. Finally, it must be taken into consideration that 2 artists does not have to be considered mutually similar. i.e. A may be similar to B, but B may be more similar to C. Since only the most [up to] 10 similar artists are taken into consideration, in some use-cases situations like those are possible. # Scoring methods Similarity scoring for genre and style weighting. There are multiple scoring methods available, i.e. how tags from a library track are compared against a reference track to retrieve similarity: - LINEAR: simple proportion, i.e. if 3/4 tags match, then its similarity score will be 75%. Thus being always "linear" no matter the tag count. - LOGARITHMIC: uses a logarithmic curve to map linear scoring to a curve. Little sensitivity to tag values counts (n). i.e. if 75% of values match, then it will output a score near 80% for all tag counts (n). - LOGISTIC: uses the logistic curve to map linear scoring. High sensitivity to tag values counts (n). i.e. if 75% of values match, then it will output a score from ~80% to ~95% depending on tag counts (n). - NORMAL: uses the cumulative distribution function of a normal distribution to map linear scoring. Medium sensitivity to tag values counts (n). i.e. if 50% of values match, then it will output a score from ~70% to ~90% depending on tag counts (n). S-shaped, giving more weight to high scores. 'LOGARITHMIC' and 'LOGISTIC' scoring methods take into account that tracks with a lot of genre/style tags don't return so many matches because it's almost impossible to match all of them. Therefore a curve is used, giving an extra weight to lower matches, specially for high tag values counts (n). 'NORMAL' scoring method just acts like a high-pass filter for scoring, where the turning point is set at 40%. High tag value counts just make it smoother. This method considers that similar tracks follow a normal distribution where the most similar ones will all be over the 40% simil. critical point. A chart comparing the available methods can be found at: '.\helpers\readme\search_by_distance_scoring.png' # Scoring methods: chart (remove code formatting to see image) `![Chart](https://github.com/regorxxx/Search-by-Distance-SMP/blob/main/helpers/readme/search_by_distance_scoring.png?raw=true "Chart")` # Sorting: smart shuffle Shuffles tracks according to tags (for ex. artist) in a semi-random pattern, ensuring no 2 consecutive tracks have the same tag and also that they are distributed in an homogeneous way. Differs from intercalation in the way tracks are ordered (without strict alternation), i.e. it doesn't follow a pattern (ABCABCAA) when it's possible to ensure no A tracks are together (CABACABA). Follows [Spotify design](https://engineering.atspotify.com/2014/02/how-to-shuffle-songs/). Contrary to [Spotify's preferences to recently played/loved tracks](https://thetab.com/uk/2021/11/17/spotify-shuffle-explained-228639), this algorithm is truly "random", by default, in the sense that there is no preference for any track, it just ensures artists are distributed evenly with some randomness (so they don't appear in clusters). The sorting bias may be configured to prioritize tracks by play count, rating, popularity, last payed or a custom TitleFormat expression: - Play count: requires foo_playcount or foo_enhanced_playcount. TF: `$max(%PLAY_COUNT%,%LASTFM_PLAY_COUNT%,0)` - Rating: requires foo_playcount. TF: `$max(%RATING%,$meta(RATING),0)` - Popularity: requires Find & Play SMP package and library tagged with Last\.fm statistics (for more support check Find & Play's hydrogenaud.io forum thread). TF: `$max($meta(Track Statistics Last.fm,5[score]),0)` - Last played: requires foo_playcount or foo_enhanced_playcount. TF: `%LAST_PLAYED_ENHANCED% | %LAST_PLAYED%` - Custom TF: requirements associated to expression/tags. There is an additional option to also try to avoid consecutive tracks with these conditions: - Instrumental tracks: (any of both) GENRE|STYLE|FOLKSONOMY = instrumental LANGUAGE = zxx - Live tracks: GENRE|STYLE|FOLKSONOMY = live - Female/male vocals tracks: GENRE|STYLE|FOLKSONOMY = female vocal These rules apply in addition to the main smart shuffle, swapping tracks position whenever possible without altering the main logic. # Sorting: harmonic mixing DJ-like playlist creation with key changes following harmonic mixing rules... contiguous tracks in the mix are most often either in the same key, or their keys are relative or in a subdominant or dominant relationship with one another. The primary goal is to create a smooth transition between songs, creating an harmonious and consonant mash-up with any music genre. Uses 9 movements described at 'camelotWheel' on camelot_wheel_xxx.js The movements creates a 'path' along the track keys, so even changing or skipping one movement changes drastically the path. Therefore, the track selection may change on every execution. Specially if there are not tracks on the pool to match all required movements. Those unmatched movements will get skipped (lowering the playlist length per step), but next movements are relative to the currently selected track... so successive calls on a 'small' pool, will give totally different playlist lengths. We are not matching only keys, but a 'key path', which is stricter. # Recipes & Themes Themes are 'tag containers' which emulate track's tags, to be used with 'Search by Distance' buttons and tools instead of evaluating with the selected track. Recipes are 'variables containers' which emulate the args of 'Search by Distance' buttons and tools instead of using hard-coded variables or properties. Note recipes may also contain a forced theme, either the entire object or a link to a theme file (using its path or filename). Both may be used in buttons, tools or with Playlist Tools: - Pools presets may link to these files in their arguments. - Search by GRAPH\WEIGHT\DYNGENRE menu presets may link to these files in their arguments to have menus which use the variables set on those files instead of hard-coded in the menu (working similar to custom buttons). - Custom buttons may use recipes and themes to tweak their behavior (see tooltip). Theme files may be created using the examples files, buttons or Playlist tools. - When using Playlist Tools there is a menu entry (see Configuration\Search by Distance) to use the selected track's tags as reference and save them to a json file directly. In that case, the tags names put at the properties panel will be used. i.e. you can remap genre to use 'GENRE' and 'ALL MUSIC GENRE', etc. - When using custom buttons just use the theme menu (Shift + Left Click). Same comments apply. Both type of files may be set to 'Hidden' (file attribute) to not show them on the menus, instead of deleting them. Some files, meant for internal use only, may be hidden by default. Although all recipes are shown by default in a clean installation, it's recommended to hide those which will not be used on a daily basis to clean up the menu using the tip above. # User descriptors Custom genre/styles, substitutions or other variables may be added to the graph by the user using the existing framework. Instead of editing the main file, you can add any edit to an user configurable file which should be placed at the profile folder on: 'js_data/helpers/music_graph_descriptors_xxx_user.js'. This is done to ensure the file does not get overwritten when updating the scripts. Check sample at helpers folder for more info and copy it to that path. Alternatively the custom button has an entry to open the user descriptors which may be used to open it. If the file does not exist yet, a copy will be created and then opened. It's irrelevant whether you add your changes to the original file or the user's one but note on future script updates the main file may be updated too and therefore changes would be lost. Both the html and foobar scripts will use any setting on the user file (as if it were in the main file), so there is no other difference. Anything at the docs which points to one of the file applies to the other [scripts folder] 'helpers/music_graph_descriptors_xxx.js' [profile folder] 'js_data/helpers/music_graph_descriptors_xxx_user.js'. # Tagging requisites All the tools use this universal tag structure: TAGS NOTATION | Description | Tag name | Tag values (examples) | Notes / Allowed | |--------------------|----------------|-----------------------|-------------------------------| | Main genres | GENRE | Alt. Rock | Multivalue | | Sub-genres | STYLE | 80s Rock; Pop Rock | Multivalue | | Moods | MOOD | Not acoustic; Catchy | Multivalue | | Themes | THEME | Reflection; Summer | Multivalue | | Key (any notation) | KEY | Am | Open, Camelot or Standard key | | Beats per Minute | BPM | 95 | Single value | | Composers | COMPOSER | Jimi Hendrix | Multivalue | | Year | DATE | 1964 | Preferred in year format | | Artists | ARTIST | Lauryn Hill; Sade | Multivalue | | Track title | TITLE | Amber glow | Single value | | Featured artists | INVOLVEDPEOPLE | Natalia M. King | Multivalue | | Rating | RATING | 3 | Plugin or Tag | Rationale: It's not feasible to support arbitrary tag names when there are a dozen of tools using their own. Also tag names should use standard names and not proprietary names. Proprietary tags like "GENRE LAST\.FM" are only meant as intermediary tags which should be renamed, remapped or merged. If you use other tag names then you have these options: 1. Rename your tags. 2. Clone your tags and use names at top. 3. Configure scripts to remap standard tags to your own tags. 4. Configure your tagging tools to output proper tag names. WARNING: All tags will be split by comma into multiple values. This is specially important for 'genre' or 'style' tags, since values like: `Folk, World, & Country` will be split into `Folk`, `World`, `& Country`. Tag properly your files to avoid this situation. NOTE: See [Global tag Remap](#Tags-global-remap) readme for permanent remapping in all tools. -------------------------------------------------------------------------- Example ------- I have used the BIO panel to tag my files, and they are all tagged with things like: "ALBUM GENRE ALLMUSIC", "ALBUM GENRE LAST\.FM", "ALBUM THEME ALLMUSIC", ... I also have my own genre tag named as "MY GENRE". Finally I have retrieved keys for my tracks using Traktor and the tag is written as "INITIAL KEY". - GENRE: (1) Is not possible for genres, since I intend to use all of them (3 tags). (2) Would be possible, Masstagger for ex. allows to merge different tags into one new field. I could simply retag all my tracks this way with a new tag name "GENRE" which contains the values of the other 3. (3) Probably the best option. For ex. Playlist Tools or Search by Distance already allow to remap any group of tags to GENRE, merging their values and removing duplicates, without touching the files. I will have to look for "Tag remapping" at the associated buttons in their config. (4) Picard allows to use scripting to set tag names before writing tags to files. This is obviously meant for these use-cases where you have different sources for a tag meant to be the same (genre on All Music, Last Fm, ...) and they should be merged or properly named. Same comment applies for other tools. BIO may be configured too, for ex. "ALBUM THEME ALLMUSIC" could be saved as just "THEME" if that's the only tag source used. - KEY: (1) "INITIALL KEY" may be renamed to "KEY" for all my files. (2) Idem. Trivial to clone single tag as "Key". (3) Idem. Remapping would allow to use the tag 'as is' without touching the original files or rewriting all tags. (4) It would be easier if from now on if I configure Traktor to write tag as "KEY". # Tags sources Tags may be sourced from: TAGS SOURCES | Tag name | Software (UNIX/Windows) | |----------|---------------------------------------------------------| | KEY | Foobar - Playlist Tools script (**) https://github.com/regorxxx/Playlist-Tools-SMP Picard - AcousticBrainz Tags plugin https://picard.musicbrainz.org/plugins/ KeyFinder (*) https://github.com/ibsh/is_KeyFinder | | BPM | Foobar - Playlist Tools script (**) https://github.com/regorxxx/Playlist-Tools-SMP Picard - AcousticBrainz Tags plugin https://picard.musicbrainz.org/plugins/ KeyFinder (*) https://github.com/ibsh/is_KeyFinder Foobar - Foo_BPM Plugin https://www.foobar2000.org/components/view/foo_bpm | | MOOD | Picard - AcousticBrainz Tags plugin https://picard.musicbrainz.org/plugins/ Foobar - Biography script (*) https://github.com/Wil-B/Biography | | THEME | Foobar - Biography script (*) https://github.com/Wil-B/Biography | | COMPOSER | Picard https://picard.musicbrainz.org/downloads/ | | LOCALE | Foobar - World Map script https://github.com/regorxxx/World-Map-SMP Foobar - Biography script (*) https://github.com/Wil-B/Biography | (*) May require additional configuration or steps to either set a compatible tag name or merge values. See [Tagging requisites](#Tagging-requisites) readme. (**) Outputs same values than Picard / AcousticBrainz. i.e. fully compatible (see warning below). NOTES AND WARNINGS: - GENRE and STYLE tags from Discogs or other sources: Some tagging sources offer conglomerate of genre/styles as a single value. For ex. `Folk, World, & Country`. For all purposes, GENRE and STYLE tags are usually split into multiple values by comma, so it will be split into `Folk`, `World` and `& Country`. In such case, tag may be remapped to something like `$replace(%GENRE%,', &',',')`. Otherwise tag properly your files, splitting the values into real genre/styles instead of conglomerates to avoid this situation or just don't use those values on tools like 'Search by distance', which would made no sense. - Getting KEY tag from different sources: Different software use different algorithms to compute KEY, therefore it is not recommended to mix tagging from different sources. Stick with one for the entire library; that way even if the KEY is wrong (not the real key), the errors will be the same across the entire library (essential for comparison purposes). - Mixx: https://mixxx.org Provides KEY tags, but will flatten all multivalue tags with spaces, making impossible to undone the tagging. Issue has been unresolved for years. - Essentia: https://essentia.upf.edu/download.html See Precompiled extractor binaries (Linux, OSX, Win). Allow to extract KEY and BPM tags, but output to a txt file. Could be used with scripting for batch processing: `streaming_extractor_music.exe file.flac out.txt` Playlist Tools script (see at top) has a tagger which may use it to tag files on batch too. # Other tags notes Some additional notes for tags apply: TAGS TECHNICAL NOTES | Tag name | INTERNALS | Notes / Allowed | |------------|----------------------|------------------------------| | STYLEGENRE | [...STYLE; ...GENRE] | Merged from standard tags | | KEY | Am | 8A | 1m | All are allowed | | DATE | $YEAR(%DATE%) | Used by default with this TF | - STYLEGENRE: is a virtual tag used for internal use, retrieved from STYLE and GENRE standard tags. See [Tagging requisites](#Tagging-requisites) readme. - KEY: may use Camelot notation (8A), Open Key notation (1m) or Standard notation (Am). Programs like Picard offer standard notation by default. There is no need to change the format in any of the tools, since all three are supported (ex. harmonic mixing). - DATE: as noted on [Tagging requisites](#Tagging-requisites) readme, year format is preferred. In any case, at most places the DATE tag is rewrapped to use the TF to retrieve only the year part. Usage of a different format is at your own risk (although it should work). # Remove duplicates Removes tracks with same tags (also real duplicated files) from playlists. It's superior to foobar2000 functionality since it checks more things than just having the same exact file multiple times... Note you can use these functions to filter lists! i.e. if you only check by artist/date, then any track with same artist/date is considered duplicated. That means this could be used both to find duplicates or for custom post-playlist creation filtering (1 track per artist, 1 track per date, etc.) Usage on playlist tools: - Tags and n+1 var may be set via menus. - See comments below (buttons) for usage. It applies the same. Usage on buttons: - Tags or TF expressions may be set via menus. There are 3 possible fields, but multiple tags may also be set in the same field. (`%ARTIST% - %TITLE% - %DATE%` is equivalent to setting them separately on the 3 properties) - When adding a tag on a field, '%' may be omitted. When adding multiple tags or TF expressions, you must add all TF symbols (% and $). - Setting the n+1 duplicates allowed variable is used for the filtering button. A value of 0 makes it work as the remove duplicates button (no duplicates allowed). A value of 1, would allow 2 (1+1) tracks with the same set tags... - Since the tags are configurable, the concept of 'duplicate' is related to those tags... that's why this tool may also be used to filter playlists instead of removing real duplicates too. **Note on 'Advanced RegExp title matching':** Matching duplicates by %TITLE% may be somewhat tricky as soon as a library contains multiple versions of the same track. For ex: live, acoustic, remix versions, etc. To account for these, RegExp may be used, to allow partial matching to tracks with things like '(acoustic)' or '(2022 remix)' on the title. Obviously these are no real 'duplicates', but the philosophy behind the 'remove duplicates' concept is not having 2 times the same song on a playlist, so having multiple versions of the same track is undesirable in many cases. This feature can be selectively enabled or disabled. # Scatter by tags Reorders selection to avoid consecutive tracks with the same configurable tag. Can be used to scatter instrumental tracks, an specific genre, mood, etc. Whenever the selection has so many tracks with that tag value, than scattering them would not allow having at least one different valued track between them, it's left as is. The intercalate option tries to avoid consecutive tags with same genre, style, artist, ... thus not tied to an specific tag value. It the list have 10 different artists, it tries to reorder the tracks to have those 10 artists on every subset of 10 tracks. Whenever that's not possible, intercalation is still applied, having tracks with consecutive same tag at the end. For ex (artist): A, B, C, A, B, C, A, B, A, A # Global settings Additional settings applicable to all scripts may be found at: AppData\Roaming\foobar2000\js_data\presets\global\*.json (or profile folder in portable installations) Such 'global' folder contains multiple *.json files which expose different global settings shared by all scripts. For ex. the default fonts used, the tooltip font, update checking, etc. To edit them, just open the file in any text editor and read the descriptions and variables. They should be self explanatory. Note the panels must be reloaded (or foobar2000 restarted) to apply the new settings. Also note in some cases these settings are the default values used the first time a panel is installed... i.e. changing them afterwards will not change such settings on an already panel. This usually happens when that setting can also be customized within the panel. For ex. the tooltip font is applied after reload/restart, since it is not configurable via panels, but the default tags set at 'globTags.json' are applied in most cases once at installation. Since scripts like Playlist-Tools-SMP or Search-By-Distance-SMP allow to configure tags and queries, those are only the values used by default. But once installed, they can only be changed via menus (or editing the properties panel). # Tags global remap All tools use a default set of tags, pretty standard, remapped like: - Date → `$year(%DATE%)` - Artist → `ARTIST` - Genre → `GENRE` - Style → `STYLE` - Mood → `MOOD` - Bpm → `BPM` - Key → `KEY` - Rating → `%RATING%` There are many more! For a comprehensive list check: [FOOBAR_PROFILE]\js_data\presets\global\globTags.json By default all tools will tend to use such associations whenever they are not configured to use other tags or remapping. Many tools have specific customizable remapping options (for ex. 'Search by Distance' tools), while others -which use queries- are simply easily editable manually. In case you want to globally remap a tag, without having to manually edit the configuration of every tool... the files found at the folder at top can be edited. In such case every new script installed will automatically load those values, and existing tools will require a 'Restore defaults', 'Reset configuration' or similar action to reload them. Note your personalized settings may be lost on the process, so it should be done as soon as possible when installing scripts for the first time. This may come handy for tools which involve queries, since those are usually set on the scripts their-selves and can not be set in a way compatible with every user's tag patterns. This patches that hole. **Example:** Let's say in our library the date of a track is associated to 'ORIGINAL RELEASE DATE' instead of 'DATE' tag (not recommended, such tagging pattern will break compatibility with most software). If we want to remap all date usage to our custom tag, just edit 'globTags.json' and change: ``` [...] "date": "$year(%DATE%)", [...] ``` To: ``` [...] "date": "$year(%ORIGINAL RELEASE DATE%)", [...] ``` # Toolbar Toolbar panel allows to fully configure sub-modules loaded as buttons. R. Click to open toolbar menu: - Add new buttons - Remove - Change order - Restore defaults - Change colors - Change text size - Change orientation and buttons wrap Hold R. Click to reorder buttons. M. Click to show headless buttons for 2 seconds. Then they are hidden. Shift + Win + R. Click to open SMP/JSplitter panel menu (works globally on any script and panel, at any position). The button list is automatically built from this path: '.\buttons\*.js' Some buttons can be 'duplicated'. i.e. having multiple instances with different settings. Whenever that's not possible, menu entry will be greyed if there is already copy of the same button on the toolbar. Finally, some buttons may require another one to be loaded first... it's also indicated in the related entry if the requirement is not met. Once added, check their tooltips to find relevant info about their usage, settings, mouse + key modifiers, etc. Buttons which are meant to be configured using the properties panel have their prefix added to the tooltip text ('sbd0', 'rd0', ...) to make it easier to find their associated properties. Note every button has its own configuration, they are NOT shared. i.e. if you add 3 buttons of the same tool, everyone must be configured separately (for ex. for tag remapping).