Article

2023-10-31 --Tech & Moderator Notes

When I started the Burg, 22 years ago, my original idea was to take smaller excerpts from news stories to follow what was going on with the Wonderful World of Terror. I registered the domain Thugburg at the same time, thinking I could keep track of the Bad Guy players and their ever-sifting organizational fronts.

Silly me. One person only has so much attention span. That's why we invented division of labor in my younger days, back in the Upper Paleolithic.

I still think it was a good idea. I covered the organizational thing and the big turban tracking, kinda sorta, with the translation routine. (Thank you, Righthaven, you rat toad sonsabitches!) It's not a good solution to me though. For years we've had the presentation of events half-hidden under big yellow repetitive blotches. It's not even databased; it's a great big brute force Javascript include.

There's gotta be a better way.

My thought at the moment is still half-manual. Build a Postgres table with the orgs we're interested in. Then fill in the organization's membership as it pops up. That still leaves us with the problem of associating the name and the org. We had separate org and name links before and that didn't work well. For one thing, we're also interested in non-WoT names, like who's the UN Human Rights head cheese? Who's the head of UNRWA?

Then we'd somehow have to tie Obits into it (Thank you, Seal Team Six and IDF!) We should probably also be able to generate an on-demand report on who died that we've been interested in during a specific time period.

I've only got vague ideas on how to do any of these that will work and be convenient to mods. I think those are the only two major constraints. Anybody else with any good ideas?

Posted by Fred 2023-10-31 00:00|| || Front Page|| [11133 views ] Top

#1 Here is what ChatGPT has to say on the matter:

It seems like you're discussing your thoughts on the development and management of a website or platform focused on tracking various organizations and individuals related to terrorism and other relevant news. Your initial idea of tracking information about the "Wonderful World of Terror" is indeed important for security and awareness purposes, but it's also a challenging endeavor. Here are some thoughts and suggestions on your ideas:

Database Structure: Using a Postgres database is a good start. You can create tables for organizations, individuals, and events. It's essential to have well-structured and normalized tables to efficiently store and retrieve data.

Data Collection: Gathering data can be a mix of manual and automated processes. You can have moderators or contributors who manually input data about organizations, individuals, and their affiliations. Automation can help in data gathering from public sources, but this might require more advanced web scraping and data extraction techniques.

Entity Linking: Associating names with organizations and individuals can be challenging. Natural language processing (NLP) techniques can be used to extract and link entities. Named Entity Recognition (NER) tools can help identify and categorize names, organizations, and other entities within text.

Obituary Tracking: Tracking deaths and obituaries is crucial for keeping your database up to date. You can automate this process by monitoring news sources for obituaries related to the individuals or organizations you're interested in.

Reporting: Developing a reporting system for users to generate on-demand reports is important. You can use query tools that allow users to select criteria like time period, type of event, or specific entities of interest.

Moderation: Ensure that you have a moderation system in place to verify the data entered by contributors. This will help maintain the accuracy and quality of the information in your database.

User Interface: Consider creating a user-friendly web interface for both moderators and end-users to interact with the data.

Collaboration: You might want to collaborate with academic institutions, think tanks, or government agencies working on similar projects. Sharing resources and data can be beneficial.

Privacy and Ethical Considerations: Be mindful of privacy and ethical concerns when collecting and sharing data related to individuals, especially in the context of counterterrorism efforts.

Scalability: As your database grows, ensure that your infrastructure can handle the increasing volume of data efficiently.

Data Security: Implement robust security measures to protect the data you're collecting and the identity of your contributors.

Legal Considerations: Ensure you are compliant with data privacy laws and regulations in your jurisdiction.

Developing and maintaining such a platform is a complex task, but it can provide valuable insights and awareness in the field you're interested in. Collaboration with experts and organizations in counterterrorism and security fields may also offer additional guidance and resources.

Posted by badanov 2023-10-31 00:06|| 2023-10-31 00:06|| Front Page Top

#2 Databases are little more than containers. You should automate the scraping of data, and placing terrorists and organizations in those containers, and then teach your scraper to look for spelling variations, which seems to be the issues with data acquisition.

Posted by badanov 2023-10-31 00:31|| 2023-10-31 00:31|| Front Page Top

#3 Moderation: Ensure that you have a moderation system in place to verify the data entered by contributors.

We rely on the group to challenge information contributed by both moderators and by Rantburgers. The variety of knowledge and experiences the individuals of the group bring to the table is a constant education and delight — not that the AI cares about delight. But just yesterday we got the answer to whether a certain DA in California was a Soros protégé, which has been added to the searchable archive.

Posted by trailing wife 2023-10-31 00:39|| 2023-10-31 00:39|| Front Page Top

#4 I am not php/apache/mysql/Linux. I am 100% Microsoft Windows dev.

You should consider researching and building AI capabilities. Big names like TensorFlow (open source), PyTorch, and scikit-learn (its free) offer APIs that PHP can embrace.

Its another hill to climb, but I understand the benefits of letting AI do the work gives you phenomenal results that can be realized fairly quickly with a little dedication.

Numerous AI communities that you can hook up with:
https://www.analyticsinsight.net/top-10-ai-communities-for-ai-enthusiasts/

Posted by Enver Dingle4916 2023-10-31 01:04|| 2023-10-31 01:04|| Front Page Top

#5 Thugburg could be rather simple. A directed graph with names and linkages. Start with the focal centres of evil in the world, Soros and Iran, maybe Putin and Xi. The rest will eventually link to them.

Posted by Sheba Thease8880 2023-10-31 04:20|| 2023-10-31 04:20|| Front Page Top

#6 I recommend my old favorite, Analyst Note book or ANB.

Social network analysis search tools be fun.

Posted by Besoeker 2023-10-31 04:39|| 2023-10-31 04:39|| Front Page Top

#7 A directed graph with names and linkages.

I second this, however...
Don't be a victim of 'technology in search of a problem to solve'. 22 years of data is a lot to risk in a platform upgrade.

Had I a say, what I would like is a time-series, event geo-tracker which draws incidents from your existing data store. That product would contribute to building a chronological graph of actors&victims(nodes) and geographical events(edges) used to exploit some nexus of influences.

EX:
Iran can no longer fly arms to Aleppo, what is their alternative? What forces, when, contributed to securing that secondary site? Where are those forces currently active and are can we project new distribution points?

Use the AI [if you must] in a supervised learning mode to backprop train against your existing 22 years of data.

Posted by Skidmark 2023-10-31 05:53|| 2023-10-31 05:53|| Front Page Top

#8 What are the questions you want the system to be able to answer? (Who is alive associated with X-named org, and what alternative names do they have? What are the current and previous alliances of X-named org? Does name Y match any names we are tracking, and if so which ones? etc)

Maybe designing the specs first might guide figuring out what you need.

Posted by James 2023-10-31 12:17|| https://idontknowbut.blogspot.com 2023-10-31 12:17|| Front Page Top

#9 There is no intelligence at all involved at this point. It is high speed search using algorithms that look at previous searches.

Posted by M. Murcek 2023-10-31 12:21|| 2023-10-31 12:21|| Front Page Top

#10 When all else fails, 3x5 notecards, pushpins and yarn? Acetate overlay drops on a mapboard?
That pretty much plumbed the depths of my technical expertise!

Posted by NoMoreBS 2023-10-31 13:11|| 2023-10-31 13:11|| Front Page Top

#11 Sounds like you're trying to duplicate the efforts of the Klingons. But with the way they are, you might perform a valuable service.

Posted by Thrineth Shuger1620 2023-10-31 13:51|| 2023-10-31 13:51|| Front Page Top

#12 LOL, ChatGPT the new Mr. Obvious.

I agree on the overall design being a directed graph. But, the data you've collected over the years is your valuable asset. I'd say two database tables at minimum: organizations and individuals. Possibly another one for locations.

The graph edges would be the (two-way) links between individuals and organizations. Individuals can link to several organizations and vice-versa.

Once your data is safely in the database, the path to developing edit and display routines should be pretty clear. You might also consider accessing the data via an API instead of directly - might be better for web access.

Posted by KBK 2023-10-31 15:54|| 2023-10-31 15:54|| Front Page Top

09:43 Mullah Richard
09:27 Warthog
09:11 Mercutio
09:07 AlmostAnonymous5839
08:52 Matt
08:24 Matt
08:20 SteveS
07:43 Procopius2k
07:42 BrerRabbit
07:42 Procopius2k
07:39 Procopius2k
07:36 Procopius2k
07:35 Procopius2k
07:34 trailing wife
07:31 Procopius2k
07:30 NN2N1
07:22 NN2N1
07:18 trailing wife
07:14 Richard Aubrey
07:10 NN2N1
07:09 Besoeker
07:03 NN2N1
06:58 NN2N1
06:58 Besoeker