/make/ - Creative

World made of things

Want your event posted here? Requests accepted in this /meta/ thread.

Max message length: 5120

Drag files to upload or
click here to select them

Maximum 5 files / Maximum size: 20.00 MB

More

(used to delete files and postings)


Programming/dev thread Anonymous 01/28/2020 (Tue) 16:45:57 No. 3
if (bodyCannotBeEmpty) {
return getShittyOP();
}
else {
return "";
}
(4.94 MB 1194x854 2020-01-29_17-29-21.webm)
I just had an idea for how to improve the console in eitochan; by listing all the commands that are relevant to what you've typed.
Dammit, I was hoping sage wouldn't bump the board in the webring so I could blogpost about stupid shit without bumping the board to the top. It doesn't seem to have an effect though.
Open file (2.44 MB 998x914 reply.webm)
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA FUCK So the reason I've been having so much trouble with databases and getting depressed and demotivated this entire time is that the node.js module is a mother fucking piece of shit that BREAKS THE SYNTAX BY DEFAULT. Basically you'll just get syntax errors for valid queries, even when I literally copypasted the exact thing from MariaDB documentation 1:1, it still gave me syntax errors and I just could not figure out why (because SQL databases are fucking genius and don't tell you what was wrong about it despite knowing that there's a problem and where it is), and was unable to do anything smart because of this. If you don't want that to happen, you have to explicitly go out of your way to tell the module not to break your shit. Now there's only(lol) 2 problems that are caused purely by this shitty fucking cancerous ecosystem of pajeet nigger shit. The first is that javascript Date object is an absolutely unworkable failure of software design and makes it nearly impossible to handle dates in UTF correctly, and it keeps randomly and turning between strings and Date objects when I get a date from the database. It mostly works but the board list currently crashes because of it. The second is that I can't get the database to accept Unicode characters. Once again I'm literally copypasting what people are telling me to do and it doesn't fucking do shit. Anyway, the good thing is that all the central database related things are now working well, and I can finally think more concretely about how to use the database. There used to be a problem where the post counters may become incorrect because I did weird nested query loops and shit. But thanks to the database's new ability work like it's meant to, everything is now simpler and better, and I just finished reply link parsing. Reply links show (OP) if the target is thread OP, and → if the target is in another thread (or at least they will once the database lets me add unicode characters). Backlinks are also displayed without scripts. Also none of that requires scripts. Webm related. Post formatting isn't actually that great internally since I did it in javascript, but it mostly works, the main issue is that other tags will apply inside of inlined code tags (or monospace text as some people call it). I'd rather parse the post in C but I haven't been able to figure out how to make C++ plugins work, that thing is another total disaster on top of the pile. There's like 4 different methods for making them and they're all a complete mess.
>>9 that's correct. the current webring api 'bumps' for board activity, and atm, any activity registers as such, regardless of sage. it doesn't have to work this way, and i even brought this topic up but no one in charge seems to care about the issue afaict.
>>27 >but I haven't been able to figure out how to make C++ plugins work i had to integrate node + C++ in a project i had 2 years ago. the c++ bit is easy, as everything is standard. getting javascript to talk properly was more of a challenge. btw, i'd recommend you use VS Code as the editor for such work, as it pretty smoothly integrates both languages into an umbrella project. https://nodejs.org/api/addons.html
>>32 I'm definitely having sage also prevent webring bumps in Nchan. I've also thought about having a "supersage" or something instead, so you can differentiate from sageing "politely", but I'm not sure if it's necessary. >>33 The APIs they've set up for interfacing between Node and C++ are just a mess (probably just because javascript itself is a mess), I'm sure I could get it to work with a little experimentation, but the bigger problem is their build setup. I'm used to just calling a simple batch file or shell script that calls the compiler, so I don't have patience for installing 17 different things and following a list of instructions that fail at step 2 even though I followed them verbatim. That happened when I got annoyed at Node for the millionth time and wanted to redo the whole server on top of libuv in C, that actually sounds much more interesting to me right now than C++ plugins for Node.js. >VS Code I use Sublime and don't really feel like I need anything more. I just want it to be simple and light and have a few things I'm used to, and not have all kinds of annoying interruptive shit that people like for some reason. I have to use Wordpress at work and the text editor makes me want to fucking murder someone because it's constantly pasting shit in the middle of my code and doing weird actions when I try to press the arrow keys to move the text cursor, and popping up errors about incorrect syntax when I'm still in the middle of typing it. I have a very low tolerance for anything like that.
>>34 i've been working on some software not long ago and thinking about expanding it out into a server. it's pure C++ with a thin wrapper over libcurl for networking. with the advances in C++17 and the new ones coming up in C++20, i think i'll see if i can just do the whole thing in native, standard code. >I have a very low tolerance for anything like that. heh i would too. vs code works rather well, though it's not lightweight, particularly. it's certainly pro-grade in general however. i don't use it all the time but appreciate it when i do.
>>34 >libuv >>36 btw, i plan to work with the asio library for my server plans. it very well wrung-out, and serves as the basis for the upcoming networking standard in C++. here's a non-boost, standalone version of the library. https://github.com/chriskohlhoff/asio
>>36 I'm not personally a fan of anything C++, though I don't mind it since I can just write my own code like I would in C. There's definitely some things I'd like to have in my editor, like some kind of highlight that shows the function definition when I'm using the function somewhere. I tend to forget what the exact arguments are and which order they're in. I'd want it to be just for show and completely noninteractable though.
Open file (101.99 KB 802x630 screenshot4b.png)
>>38 >ike some kind of highlight that shows the function definition when I'm using the function somewhere. m$ has coined the term 'intellisense' for this concept. clang has implemented a rather sophisticated AST that handles this type of parsing (a very complex problem, actually). juci++ implements it, and i can recommend it as a good C/C++ editor, been using it for years. vs code also provides this capability with the addition of a couple of C++ addons. in either solution, C code 'intellisense' will work fine.
>>39 Well I mean, Sublime has plugins that do it too, and it can show a name-only list without plugins, but the problem is that I can't get it to stop hijacking keyboard keys. I don't want it to do anything no matter what key I press, I only want to see it.
Open file (30.46 KB 878x508 Untitled.png)
Time for some old fashioned blogposting™ >get tired of Javascript and Nodejs and the whole ecosystem around them (again) >make some prototypes of Nchan's internal functions in C for fun >I've arrived in the promised land >want to make my own server from scratch (again) >explore libuv for a while >decide that I don't want anything to do with that piece of shit either >realize that Libuv, rather than being some very cool and advanced system, is probably just a wrapper around IOCP (Windows) and epoll (Linux) anyway, with stupid shit added on top and inbetween and making it harder to compile for no reason So lately I've been trying to figure out how to make a server from scratch using no libraries except for the operating system's networking mechanisms. I've always had trouble understanding that shit because it's all so convoluted and there's no very good guides for them, particularly the more advanced systems that are good at handling high amounts of traffic (e.g. IOCP). I experimented with epoll for a while, but my Linux laptop is such a pain to use that I decided to tackle IOCP on Windows first since that's what my main computer uses. After cleaning up Microsoft's overly complicated example and removing all error handling, it's down to about 500 lines of code. I managed to respond to my browser's request and send some html page. I have no idea if it's practical though, because I think the data is sent/received in multiple batches if it's too big. And this is what I always struggle to understand; how do you connect what data goes where and when and who is it related to. Rather than just having some list of sockets and reading/writing their status/data or something straightforward, you always have to use these incredibly confusing methods/macros to send weird complicated structures left and right and shit. And to make matters worse, there's pointers to structures that are being cast to completely different structures in the example. It defines a custom structure with some information, and then it sends a pointer to that into some windows function that expects a different, windows specific structure. How the fuck is that supposed to work? Epoll on Linux seemed mostly simpler, but it had confusing parts too. The difficulty of making the server itself is trivial compared to the difficulty of understanding how to use these shitty networking mechanisms. I just need to get to the point where I can abstract the system for myself (where I can associate my own socket data to each thing that's received and reason about how it's being handled) and the rest should be a cakewalk.
Open file (1.97 MB 1346x740 2020-04-11_20-31-36.webm)
Well then. I managed to create a situation where the program seemingly gets stuck (it stops printing anything), but somehow still sends new pages to the browser. All I did was make the message longer and this happened. I was hoping to make progress but it's just getting less understandable.
>>56 Nevermind, turns out that this is in the thread-safe printf function that the example had. This example code is from Microsoft's own samples in github, and they fucked up their print function. Basically it checks for 2 things before printing; a complete nonsense condition (if 0 > stringLength), and the latest error code. In other words if GetLastError() returns anything at all, it won't print.
Open file (314.10 KB 398x546 cap.webm)
Just thought of an interesting way to do captcha without scripts. Apparently links can open pages into an iframe if the iframe has a name="" and the link has the equivalent target="". Webm related is this in action, it's loading and reloading a captcha form page in an iframe, without any scripts required. The idea is that if you've already completed captcha, then it'll simply load a text that says so. If you need a daily captcha, then that will show up instead. If you fail the captcha then it'll just refresh and show a different one. The downside is that there might be more clicks per post, but I think that's a lesser evil, plus you can fix it for people who have scripts enabled anyway. I find that in many sites the captcha loads when you open the page, and by the time you type it out it's already expired, and then you have to go back a page and risk losing your post etc. And that's on top of the fact that you may fail the captcha normally, and always have to re-upload files. And on top of THAT, there's usually a daily captcha that might have expired. It's just a huge headache without scripts since you can't check/send the captcha before submitting or open a popup. Someone also mentioned loading="lazy" for images, I added that to all thumbnails and it's amazing. You can load catalogs and huge threads without having to load every single image every time. Can't tell if it will be annoying if the server is slow though, remains to be seen I guess. - On a different topic, I've been working on that stupid IOCP shit for days and I just can't get it to work. It has all kinds of problems (crashing, browser loads forever) that I can't trace down, I simply don't understand how this pile of shit is meant to work. Every online example does the same shit, I even read a tutorial that explained the system in a new way and then the sample code they gave was completely different and had components that weren't in the explanation, it was basically copypaste of the same example I've read everywhere else. It's like nobody actually understands the system, and instead they just copypaste the sample code and plug shit into it. That's not my style, I want to actually know what my code is doing. But I don't understand how they managed to make this so hard, I'm not doing anything messy or convoluted, I'm not even changing the example that much, there's too much about this system that's not explained properly. Before I had this iframe captcha idea I managed to discover a thing that might be the cause of problems. The problem is the system keeps processing sockets that have already long closed. Like it posts an event literally 8 times after it has already completed and closed. That might cause a problem because I put the socket information into an array so I can reuse the data without having to call malloc all the time, but now I have no idea how to fix this because the system doesn't appear to give me any way to tell it to stop fucking sending events to the socket that's already done, nor does it have any way for me to know when it's done processing events for it (so I can know when it's safe to reuse the data). The program was already having all kinds of problems before that so it's probably not the only issue.
Open file (96.25 KB 840x700 1579071602773.jpg)
I think it's time to give up on this fucking IOCP. I just discovered that it's posting several events into the same socket at the same time (on different threads). I mean I'm sure this is all fine and intended since the only IOCP example I've ever seen before in my life is an echo server that just shits the received data back as soon as it's received regardless of what's received. But meanwhile in real world that's a completely useless behavior for the program. This is the biggest difference between my code and the example code, because I can't just echo the data back as soon as it comes in, I need to actually put together the data that I'm receiving, wait until I have all of it, process it, and then send a response to the same person afterwards. But despite working on this for like a week now I can't figure out how to do that and I haven't been able to find any examples that do anything like it. While reading about this, I also found out you're supposed to close the completion ports manually when they're finished. The provided example code opens a new one for every socket but never closes them, the whole design makes it impossible to close them. I also don't know how to approach doing that because I have no idea what the completion port is tied to, how do I know when it's ok to close it? My only remaining hope is to make a server using the old slow socket functions and hope it gives me some kind of insight that I've been missing about sockets, or something that makes me understand this system better. I said I wouldn't blogpost much but this shit drives me up the fucking wall, because if I can't read Microsoft's mind about how this is meant to be used then the alternative is to start using fucking nodejs and javascript again, which is basically like putting a giant slimy nigger box that randomly plays fart sounds between me and the computer. Of course I could just ditch Windows and only make a Linux version of the server but a) I don't want to, and b) I'll have to do this anyway when I want to make a multiplayer videogame because nobody plays games on Linux.
It finally (almost) justwerks. So I did end up checking through an old school socket guide and immediately found out something very fundamental(?) that I didn't know about. It seems like there might actually be no way to know when the client is done sending data. I'm not sure if that's true, but the guide showed an example of multiple separate messages strung back to back, and then started talking about protocols as the solution. So I assume you're supposed to parse out the expected amount of data from the HTTP header (or whatever protocol you're expecting) as it comes in rather than relying on some piece of information that the system gives you. Basically the connection just stays open and sends/receives shit whenever it feels like and it's up to you to figure it out. The IOCP guide has several places that look like the client is done sending data, there's even comments that say so, so I just assumed there's a way to know. But more importantly, this ties together to the big realization about what I've most likely been doing wrong. It's such a disjointed combination of a lot of things going wrong and the IOCP guide being fucking retarded in 2 different ways that it's hard to describe. So the IOCP guide defines this IO info structure that you use manually to tell yourself whether the current event is sending or receiving data. It's always sent to the system along with the same socket info, I couldn't figure out why it's separate so I just merged them (it basically changes it from READ to WRITE and then echoes the data, and then changes it to READ again). Another confusing thing about it was that inside of the IO info struct, there's a WSAOVERLAPPED struct, this is what the system actually expects, but it returns the pointer to your IO struct anyway (WSAOVERLAPPED is first member of the struct so Windows still understands it even though the pointer is technically the IO info struct). The WSAOVERLAPPED itself is never used in any way in the example. In order to send a reply to the client, I used WSARecv to check if there's any more data to load, and then immediately did WSASend if there wasn't. Although this isn't necessarily a mistake entirely, it becomes a mistake because of the way the IOCP guide had set up the IO struct. Both of those function want the WSAOVERLAPPED struct (IO info). In IOCP example the IO info struct is just reused, and it's probably fine because of what it does. However if you call 2 things back to back, it creates 2 different events that are using the same IO info data. So here you're actually supposed to create a new WSAOVERLAPPED struct (or IO info) and send that (the old socket guide clued me into this since the old system just tells you what the event is as opposed to making you tell yourself). And to find out when to reply, you're supposed to read the received data as it comes in, parse the HTTP header, and then use WSASend to send a reply when you've reached the expected end (HTTP header ends in 2 new lines, and if there's body content the header will tell you how long it is). Since I tried to use WSARecv to check for end of incoming data, it just created a new event for no reason, and since the IO info was shared between different types of events that were running simultaneously on multiple threads, everything got fucked.
Open file (2.25 MB 1622x822 almostworks.webm)
>>60 So after all this, I may finally understand how the IOCP system works. I rewrote it according to my new understanding and it now works with almost 100% consistency, except for one crash that occurs randomly, I haven't been able to track that down. It seems to be related to sockets closing or something, since it often happens if I cancel the page refresh. After I get a little more experience with this and fix all the bugs, I'm probably going to write some kind of guide for this because the only example that I could find doesn't explain it properly. Of course I could still be doing everything wrong and it only works by chance somehow. As for the crash itself, the reason I'm having so much difficulty is that I can't find anything that should be able to crash (unless it's some Windows function shitting itself). The IO info and socket info are never reused or removed, so there can't in theory be a conflict between them (the array of them will run out eventually and the system stops working, but I just did that to find the problem). In webm related I notice it does a bunch of sends in a row right before crashing, but all of them are on a different socket and different IO info so they can't be conflicting. It might also be stuck somewhere else which is why nothing happens inbetween the last sends. There's always a possibility that the problem is in my own string functions or something. At this point I should just bite the bullet and use a debugger. I rarely do that because I hate having to create projects in some shitty IDE, and more importantly I use a batch file to set up my compile and I don't know how to configure that in the IDE, so I basically have to move all my files and create some separate hard coded compile setup or something, I don't even remember, I don't want to remember. I was going to start using debuggers more and learn to use them proper after I get myself RemedyBG, but I haven't had a pressing need for it yet since I can usually find any bug in a minute. This time is a lot harder because it's multithreaded and the problem is most likely the IOCP system doing something unexpected that nobody bothered to explain. I don't even know how to debug multithreaded code though, since I can't imagine you can step around in the code normally, and break points will be scattered across threads. I just wish there was a way to tell where the crash happened, all you need most of the time is to know where the problem is, it's relatively straightforward to find the cause afterwards.
Open file (3.74 MB 1586x890 werks.webm)
FUCK YEAH I managed to track down the last remaining crashes, and the primary one WAS in fact in one of my own functions. I'm using a new string system that I haven't tested much that I call "defstr" (double ended flat string), it's basically a continuous chunk of memory where you can add/remove data from either start or the end without causing re-allocations or big data moves, and it re-allocates if it needs more space (or shifts around if there's a lot of extra space in the opposite end). It's perfect for this since I can add new data to the end of the buffer, and remove data from the start when I read it. Anyway one of it's functions had a missing piece that eventually caused the reallocation to fail. The weird thing is how uncommon this problem was, I should have definitely run into it before, I even tested the functions when I made them but didn't encounter it. The second crash was much rarer and I hadn't seen it in a long time, this one actually gave a windows crash prompt which tells me it's an access violation. I found the cause of it when I started thinking of how I can start re-using old IO and Socket data again so it won't just create new ones infinitely. And as soon as I removed the line that closes sockets, the crash no longer occurs. I had to change the design of things a bit such that the socket info keeps track of how many IO operations are active on it, and is only closed when the last active IO operation is removed. So now it works very consistently, it never hangs or crashes or gets stuck or fails to send resources no matter what I try on the browser's side, it never exceeds 6 active sockets and 13 active IO operations on a single browser loading a page with 30 images, and it reverts to around 17 MB memory usage after each page refresh. I actually load images directly into memory when I send them, but I think it's possible to pipe files straight into the IOCP. That'll be required eventually since I want to minimize memory usage headaches (though I'll probably use Nginx for static files anyway). There might also be some possible exploits, for example I'm not sure if sockets ever time out on their own, so a malicious user could fill the server with passive connections. There's an option to time out the IO event, but I'll just have to see. But anyway I want to get to the actually interesting part now, which is making the application server. I can already parse the most important parts of the HTTP header to figure out whether it's a POST or GET request, and what page/file the user wants to access. I don't think there's actually anything very complicated about this, reading the headers is simple, sending files is simple, the hardest part might be to parse form data, especially file uploads (I'll also need a way to pipe the file data straight into a file rather than loading the whole thing into memory). I experimented with normal forms and multiplart forms, and the data seems pretty straightforward, though I didn't try file uploads yet. Webm related, I can already serve pages and files. Besides minor tweaks here and there (and the linux equivalent of the server), there's only 1 big obstacle before I can start working on Nchan, and that's the database. I actually have no clue how to connect to a database from a C program like this. I know how to do text commands since I've used FFMPEG from one of my programs before, but you need to actually log in to the database and stuff so I don't think it's the same at all. Anyway now I can finally stop blogposting about this.
Open file (27.47 KB 530x195 texty.png)
Finally un-died enough to get around to setting up stb font properly, and I'm surprised at how clean it looks. I thought all the letter spacings would look completely fucked, but it looks mostly consistent even at very small sizes. The letters themselves don't look too great though, but it's readable at least.

Report/Delete/Moderation Forms
Delete
Report

Captcha (required for reports and bans by board staff)

no cookies?