http://www.idkfa.com/v3 idkfa: syndicated http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=4098 I'm glad you feel that way. Other than the problem of having to display threads going to indefinitely to the right, I don't understand why absolutely everything (Facebook, Twitter, Tumblr, Google+, etc.) has chosen to go flat. Maybe our brains don't work like everybody else's. kaiden@idkfa.com (kaiden) idkfa Sat, 11 Feb 2012 14:39:29 -0900 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=4098 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=4092 I definitely prefer the way you have idkfa set up. It seems much easier to follow what is being posted. I appreciate the chance to make a comment to a post in the middle of a thread and have everything 'in context' rather than purely in chronological order. norwaygirl@idkfa.com (norwaygirl) idkfa Sat, 11 Feb 2012 12:55:12 -0900 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=4092 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=4078 Haha, thanks man. kaiden@idkfa.com (kaiden) idkfa Thu, 09 Feb 2012 15:12:20 -0900 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=4078 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=4077 IDKFA probably makes up the majority of my early use of message boards. And probably also a good portion still. The fact that as from what I can remember idkfa has always been threaded means that's how I prefer to see all message boards now. I really dislike the "flat" view for the reason that even in real conversation I prefer to be able to respond to not just the initial comment, but also the last comment, and sometimes interesting things that are said in between. So good on ya josh because I prefer how you run your shit over how pretty much everyone else does it. Probably mostly out of familiarity, but maybe also just personal style as well. Those other developers can go to hell. MrFood@idkfa.com (MrFood) idkfa Thu, 09 Feb 2012 15:09:28 -0900 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=4077 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=4075 SPDCA: There's an article I happened to come across in some of my research for this, stating the author's disdain for forums that adopt "threading" rather than "flat" displays for forum interfaces: "Something interesting happens sociologically when you don't have threading: the conversation is forced along one train of thought. People feel like they can respond to the original inquiry, or they can respond to the last post, but if they want to nitpick about the third post and there are already twenty more posts after that, it's just too late." Joel Spolsky, Jan 2001 The most compelling implication of what Spolsky says is that flat forums often reduce a topic such that it is more "conversational," rather than simply passing messages and storing them in a message tree. Users are compelled to hold a conversation, based on either the original thing brought up, or the last thing said about it, and less so the in-between. (shrugs) I don't know. I've been on a few forums. And nothing about them is conversational. Forums are a unique form of communication, at once static and dynamic, with topical direction and dimension changing regardless of whatever convention users happen to uphold. Reading without threading, to me, is like reading a book written by twenty authors, none of whom proofread or try to maintain any sense of conversational narrative. It is screaming at the Internet, and lengthening a wall of text. The reason I'm looking at this at all is I'm feeling a little outnumbered. The majority of the popular forum software out there on the Internet, either open source or commercial products, are all predominantly "flat." In addition to that, they have fairly unsophisticated viewing history mechanisms. I've been trying to bring up my boundary algorithm on their developer forums, only to find that people ask why I would come up with a system like that in the kaiden@idkfa.com (kaiden) idkfa Wed, 08 Feb 2012 03:46:16 -0900 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=4075 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=3987 Well... related? Yes and no. Yes, in that it's an example of "shit you cannot and should not try to do in Excel, yet people will try to do it anyways." No, in that my particular experience that spawned my survey and subsequent awful essay wasn't related to the aforementioned project and its use of my algorithm. kaiden@idkfa.com (kaiden) idkfa Tue, 17 Jan 2012 20:36:31 -0900 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=3987 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=3986 would the application of your idkfa algorithm at work relate to your inquiries of excel elsewhere in this forum? and your critique of it in EC? kitacek@idkfa.com (kitacek) idkfa Tue, 17 Jan 2012 20:30:02 -0900 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=3986 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=3983 SPDCA: Having trouble sleeping, so I'll write a bit about some developments regarding the adventures of this algorithm. Amazingly, I had the opportunity to use this at work. It was the case where I needed to design a tool that could consume log messages from another data source. These logs were sequentially numbered (like idkfa posts), were monotonically increasing, and came through at a very, very fast pace (say 80MB in a few seconds). My log consumer needed to be robust, fast, and in addition to the statistical features it would have from scanning the log messages, it would need to "remember" if it had seen a log message provided that a message was a) accidentally/programmatically repeated, or b) explicitly sent again by an administrator. That meant that I needed a way to accurately track the messages that would be 1) quick to access and 2) take little memory. I decided to take the method I had for idkfa and implement it using Perl. The "quick" part took about a week and a half to figure out, as I was having to continuously rewrite my code to try to keep up with the data stream. Performing a linear search was far, far too slow, even in the beginning, and would be unusable after only a short period of normal usage. By the time I was done, I was using two Red-Black trees as lower and upper bound indexes (similar to the database index I use on idkfa), as well as a few other tricks made possible by having data structures stored in memory rather than a database. Unfortunately, memory usage became a killer issue. While my algorithm could keep up and scale well as it "saw" more data, there was never any guarantee that the "gaps" in the log messages would be filled. I was collecting such a huge amount of data that even if I was missing 1 in 1000 messages, within hours my process would run out of memory by being forced to store all of the gaps. It was at that point I had to make a compromise. There was kaiden@idkfa.com (kaiden) idkfa Tue, 17 Jan 2012 03:31:45 -0900 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=3983 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=1589 Sorta funny. I was poking around, and realized that there were about 400+ posts on idkfa that I had written, but hadn't "read." I used the speedreader function of the search page to mark these as read. My page load times were cut in half. So yeah. If you're a frequent reader, but you haven't read everything, and you're noticing idkfa loading somewhat slower than normal, it might be a good idea to mark the posts you don't intend on reading as read. You can do this by logging in, navigating to http://idkfa.com/v3/search.php?terms=is%3Aunread, and clicking "Mark these posts as read." It will tell you how many posts it intends to mark, and gives you an out before it marks them. kaiden@idkfa.com (kaiden) idkfa Wed, 29 Dec 2010 02:29:07 -0900 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=1589 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=1334 I think I found the reason why people don't do this. I was noticing that idkfa was running a little slow. I looked on the server, noticed there was other stuff going on, and thought nothing of it. I noticed again this afternoon, and found that nothing else was going on. Yet idkfa was taking almost 2 seconds to load each time. Looking closer, I found that the type of query I have to use to do this is not terribly efficient unless you index heavily, and index correctly. I had three indexes, one to keep track of the user ID of the message bound, one to keep track of the lower bound, and one to keep track of the higher bound. I removed these, and added a single index that contained a concatenation of all three rows (usr, lower, upper). This improved speed by reducing load times back down below 200 ms. I guess we'll just have to watch as this scales. If you folks notice more slowdowns, don't be afraid to mention it. kaiden@idkfa.com (kaiden) idkfa Mon, 29 Nov 2010 22:37:51 -0900 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=1334 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=982 If you haven't read all of the posts.... you definitely loooooooooooose. Scrotor@idkfa.com (Scrotor) idkfa Thu, 07 Oct 2010 05:22:29 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=982 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=931 oh... hmm. Well, I definitely do jump around a lot which what I click on rather than reading posts all in order. So that makes sense, I guess. Erik - I think this means I win. Yeah, you know I'm right. CapitolZebra@idkfa.com (CapitolZebra) idkfa Mon, 04 Oct 2010 15:22:07 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=931 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=930 You view posts "sparsely," that is, you tend to view posts with non-contiguous ID numbers. Rather than storing each number, I'm storing a range that says "This user has read from post #20 to post #30." Each time you read a post near that range, I can increase the range by one. But if you view a post that's more than 2 away from that range, I have to store a new range entirely. The more posts you read, the less "sparse" your viewing history so be. That you have more ranges than anyone else just means you don't make it a point to read everything, just things you're interested in. That makes you a discerning reader. :) kaiden@idkfa.com (kaiden) idkfa Mon, 04 Oct 2010 15:16:14 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=930 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=929 oh.... What did I do that made it difficult to compress? CapitolZebra@idkfa.com (CapitolZebra) idkfa Mon, 04 Oct 2010 15:11:25 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=929 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=928 You read posts in such a fashion that it makes it hard to "compress" your viewing history. You're doing nothing wrong. kaiden@idkfa.com (kaiden) idkfa Mon, 04 Oct 2010 15:09:41 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=928 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=927 i'm confused..... CapitolZebra@idkfa.com (CapitolZebra) idkfa Mon, 04 Oct 2010 15:08:18 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=927 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=918 Ahaha you lose, Ashley!! Scrotor@idkfa.com (Scrotor) idkfa Mon, 04 Oct 2010 04:04:57 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=918 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=902 SPDCA: Being fresh in my mind, I'm researching how other boards do this. phpBB: These guys cheat a little bit. They store the following: - Whether you've marked a sub-forum as read, and the last time you "read" it, or marked it as "read." - Whether you've marked a thread as read, and the last time you "read" it, or marked it "read." They never store whether you've read a given post, as default thread display is every single post that is part of a thread (and never an individual post). This means that if you read the first page of a thread, it doesn't matter of you haven't read the subsequent or even the last page of that thread, it will be marked as thread, and you'll never know whether a new post in the thread is something you've actually read, only something that was posted since you were on one of the many pages of a thread. They get by on the fact that there will always be more replies than there will be threads, which works in most cases, I suppose. Simple Machines Forum (SMF): A little more sophisticated, but they somewhat suffer from the same problem. They store: - Whether you've "read" a board, and the ID of the latest post you read (in that board). - Whether you've "read" a sub-forum, and the ID of the latest post you read (in that sub-forum). - Whether you've "read" a thread, and the ID of the latest post you read in that thread (in that thread). - Whether you've "read" a post. They do specify each message you read, as well as the the ID of the last message at each level. This is as efficient as the phpBB method, as all you have to do is find a ID later or a date newer than the one stored for a particular user. Supposedly, when a user reads all of a thread, or a sub-forum, it marks the next level up as "read." However, if people read 99/100 posts in a thread, it will kaiden@idkfa.com (kaiden) idkfa Fri, 01 Oct 2010 20:41:43 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=902 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=885 Try again. kaiden@idkfa.com (kaiden) idkfa Fri, 01 Oct 2010 14:43:24 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=885 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=884 same thing for "myidkfa - next unread" kitacek@idkfa.com (kitacek) idkfa Fri, 01 Oct 2010 14:31:41 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=884 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=883 when i try to use the ctrl-`-n shortcut, it takes me back to this post every time. kitacek@idkfa.com (kitacek) idkfa Fri, 01 Oct 2010 14:31:12 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=883 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=863 Alright, I think this is back up. Let me know if otherwise. kaiden@idkfa.com (kaiden) idkfa Fri, 01 Oct 2010 02:35:06 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=863 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=862 Also, tag feature isn't working at the moment. kaiden@idkfa.com (kaiden) idkfa Thu, 30 Sep 2010 23:00:22 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=862 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=861 Fun stuff: - Currently ~800 "bounds" defined across all users, down from the previous 4500. - User with the most bounds (the most sparse viewing history): CapitolZebra kaiden@idkfa.com (kaiden) idkfa Thu, 30 Sep 2010 22:58:02 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=861 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=860 (sneezes) What? Who? Oh, look. Got switched over to the new system. And migrated everybody's histories over. You also now have the ability to "speedread." For any search term, you can elect to mark the results as "read." You can access this from the search feature. kaiden@idkfa.com (kaiden) idkfa Thu, 30 Sep 2010 22:48:14 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=860 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=859 I'm not necessarily smart, just that I've built such things before, and watched with hot tears in my eyes as they came crashing down around me. kaiden@idkfa.com (kaiden) idkfa Thu, 30 Sep 2010 20:07:06 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=859 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=858 this is why we have to have smart people like you fix those of us who don't know what we're talking about. now if we would only listen. kitacek@idkfa.com (kitacek) idkfa Thu, 30 Sep 2010 20:01:22 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=858 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=855 Touch every message record every time somebody creates an account? Let's say you have 1000 posts, and 100 users. Worst case is the starting case: nobody has read anything, so 1,000 * 100 = 100,000 extraneous records must be stored until a user starts viewing. Add a single user, and you have to add 1,000 extra rows, even if that user only ever logs in once and never comes back. Add a single post, and that post has to include 100 extra rows to indicate the number of users that haven't viewed it. With the viewing history boundaries, worst case: Mike's even-odd viewing, 50% of total number of posts, and will only grow as they continue to view even-odd. kaiden@idkfa.com (kaiden) idkfa Thu, 30 Sep 2010 17:30:14 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=855 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=853 as users do more, though, wouldn't the required space go down? and there couldn't be an "auto-add" script when a new user joins? on a side note, my whole half of the conversation here reminds me of dilbert comics, where i'm the clueless customer and you're one of the engineers. kitacek@idkfa.com (kitacek) idkfa Thu, 30 Sep 2010 16:58:51 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=853 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=850 I'd be needlessly preallocating space. I'd also need to perform an additional update every time a new user came along, which is a little unreasonable. kaiden@idkfa.com (kaiden) idkfa Thu, 30 Sep 2010 15:46:52 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=850 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=849 what if each new message received every user's name as an attribute in its array, and when a user reads the message, the user's name is removed from the array? kitacek@idkfa.com (kitacek) idkfa Thu, 30 Sep 2010 15:25:05 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=849 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=848 He can't read your reply. It's an even-numbered message. kitacek@idkfa.com (kitacek) idkfa Thu, 30 Sep 2010 15:22:21 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=848 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=847 You're right, would cause the same problem in the reverse direction. Though, I like that you're thinking about it. kaiden@idkfa.com (kaiden) idkfa Thu, 30 Sep 2010 14:56:57 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=847 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=846 Ha. I'd still be at 50% of the previous record usage, if so. kaiden@idkfa.com (kaiden) idkfa Thu, 30 Sep 2010 14:51:29 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=846 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=845 so, I volunteer up front that I know NOTHING about computer science. I had a thought that maybe have each array be based on the message, and not the user. That way, each message says #kaiden or #kitacek have seen it. But then, I was thinking about it a little more, and maybe that doesn't reduce the amount of records you'd have... kitacek@idkfa.com (kitacek) idkfa Thu, 30 Sep 2010 14:42:00 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=845 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=844 My new objective is to now only view odd numbered posts. EDITED FOR A TYPO - I'M PLAYING A DANGEROUS GAME, JOSH Green Man@idkfa.com (Green Man) idkfa Thu, 30 Sep 2010 14:33:35 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=844 http://www.idkfa.com/v3/v_thread.php?thread_id=843&msg_id=843 So... slight problem with idkfa's backend. Here's the problem: I store a record for each message a user sees in something like ["kaiden", "Msg #256"]. This means that for every time you view a message, a record is entered into this table, and then is left alone on subsequent visits to the same message. This is by far the easiest way to give people a "viewing history." However, it doesn't scale well. We have about 800 posts on idkfa, and about 30 users. With only that many people viewing, we already have ~4,500 records in the viewing history table. Worst case for this table (everybody logs in and views everything) would be 24,000 records. Which means that with relatively few people, and only over the course of a few months, we've already reached 18% of the worst case. The more records in this table, the longer it will take to search, and the longer it will take to calculate things like "Next Unread Post" or the "#Unseen" value in the discussion area list. I worked with this for a couple of hours last night. I think I have a good idea how to better handle this. Instead of storing records like I have above, I'll store it like ["kaiden", "Msg #256", "Msg #300"]. This will mean that user "kaiden" has already viewed the range of posts from #256 to #300, and it can be assumed that every post in between has been viewed. With some careful updates, it will be the case that over time there will be less entries in this table with the more messages a given user views. What this might mean is that in the near future, I might have to reset your viewing histories. I might have a way to keep them, but I can't promise anything. Also, post tagging might have to be postponed while I rework this system as well. What this will enable in the future is the ability to mark entire threads, or even entire items as "read," as you can in other kaiden@idkfa.com (kaiden) idkfa Thu, 30 Sep 2010 12:39:19 -0800 http://idkfa.com/v3/v_thread.php?thread_id=843&msg_id=843