I know you guys hate Trade. But come on!!!

"

Fruz wrote:

"

Novynn wrote:

We'll be raising the limit to 15,000 in an upcoming patch and investigating better methods of signaling this limitation to the user.

That's a good start, thank you.

Still, I find it a bit 'amazing' that such an issue had not been pointed out before here to be completely honest.


Would filtering the results when querying at the db level result in such a significant load increase ?

On a well indexed relationnal database, putting sorting direction makes query actually faster and adding a sort by parameter is quite easy.

But I suspect they use document database like mongodb or elastic search because they are easier to manage for replication and data coming from various part of the world which sorting might have a different impact on how you query and retrieve data especially on multiple nodes but I cant give you an accurate answer on document database as I havent played with them much.

"

ffogell wrote:

On a well indexed relationnal database, putting sorting direction makes query actually faster and adding a sort by parameter is quite easy.

But I suspect they use document database like mongodb or elastic search because they are easier to manage for replication and data coming from various part of the world which sorting might have a different impact on how you query and retrieve data especially on multiple nodes but I cant give you an accurate answer on document database as I havent played with them much.

Well, given that tripling the amount of results is going to be the solution picked instead of 'properly' sorting ....
I am pretty curious about the reason why, with all technologies I've ever used, a simple sort never has been a problem to add in my own experience, so I am assuming that there is a performance issue somewhere.

Or is increasing the limit a temporary fix and there will be a better solution in the future ?
I would like to hope so.

"

Fruz wrote:

I am pretty curious about the reason why, with all technologies I've ever used, a simple sort never has been a problem to add in my own experience, so I am assuming that there is a performance issue somewhere.

ddos. and they're getting this alot. otherwise they wouldn't use crap like cloudflare which you normally only use as a means of last resort.

Fair enough, can't say that I'm very knowledgeable about ddosing so ...

/sad

"

Fruz wrote:

Fair enough, can't say that I'm very knowledgeable about ddosing so ...

/sad

There's really not much to it. It stands for Distributed Denial of Service, which kind of explains what it's meant to do.

They use botnets to simply spam a (web)server with extreme amounts of connections simultaneously, attempting to cause a bottleneck, to essentially "choke" the server. A botnet is (large) cluster of devices infected with malicious software, often without the user's knowledge.

The main issue with DDoS attacks is that it doesn't do anything other than what users on a server normally do, except the way they do it at the same time makes it a problem. Contrary to (I guess) popular belief, it's NOT hacking! The attackers don't have access to anything on the server, though it may provide a hacker with a window of opportunity in certain specific cases.

Another issue with DDoS attacks is how easy it is to pull off. 4 years ago my ISP has had their DNS server blocked for a few hours by a group of kids age 14-17. Apparently they lagged in games or something and wanted revenge. Boasting about it on social media, however, was probably not the best idea ever. I think the authorities went easy on them, but made sure it was understood that such an attack is serious, with serious consequences, among which years of prison and potentially paying for any damage caused.

Protecting your servers against such attacks isn't easy and it isn't cheap. Monetary costs obviously, but most notably performance costs that users notice. Having to identify each and every connection made to a server in order to assess whether or not it an actual user or a bot. A bit like insurance, it'll provide a safety net when something happens, but you're paying for it all the time regardless if it does or not.

Thx, I know he base principle but I never paid much attention to it, so I'm not sure what would be the impact in our case regarding the database search :).

For example : if making a query with filtering at the database level isn't much heavier ( if at all ) than a normal one, how could ddos have any heavier effect then ?

"

Xavathos wrote:

Protecting your servers against such attacks isn't easy and it isn't cheap

That is mainly what I was referring to.

"

Fruz wrote:

Thx, I know he base principle but I never paid much attention to it, so I'm not sure what would be the impact in our case regarding the database search :).

For example : if making a query with filtering at the database level isn't much heavier ( if at all ) than a normal one, how could ddos have any heavier effect then ?

"

Xavathos wrote:

Protecting your servers against such attacks isn't easy and it isn't cheap

That is mainly what I was referring to.

To answer your question, when you query a database and currently get more than 5000 results, it stops at that number, despite there being potentially hundreds of thousands more results to show. This practically controls the time it takes to resolve each instance. Which in turn helps control DDoS attacks by limiting the "damage" each instance can cause.

It also, however, stops real users from seeing all results. It's ultimately a trade-off.

"

Xavathos wrote:

To answer your question, when you query a database and currently get more than 5000 results, it stops at that number, despite there being potentially hundreds of thousands more results to show. This practically controls the time it takes to resolve each instance. Which in turn helps control DDoS attacks by limiting the "damage" each instance can cause.

It also, however, stops real users from seeing all results. It's ultimately a trade-off.

I never suggested to remove the limit, only to filter results at the database level, before applying this limit.

Filter the results with the query, and then truncate, only after having already gotten the most relevant results.

"

Fruz wrote:

"

Xavathos wrote:

To answer your question, when you query a database and currently get more than 5000 results, it stops at that number, despite there being potentially hundreds of thousands more results to show. This practically controls the time it takes to resolve each instance. Which in turn helps control DDoS attacks by limiting the "damage" each instance can cause.

It also, however, stops real users from seeing all results. It's ultimately a trade-off.

I never suggested to remove the limit, only to filter results at the database level, before applying this limit.

Filter the results with the query, and then truncate, only after having already gotten the most relevant results.

the limit applies before the filtering. it wouldn't make sense otherwise cause the real work (causing load) the database has to do internally is in looking up all table entries and sort them.

there has to be a limit, i think poe.trade also has one but it's just set higher cause he only provides his service to english players while ggg's trade interface is international.

"

Fruz wrote:

"

Xavathos wrote:

To answer your question, when you query a database and currently get more than 5000 results, it stops at that number, despite there being potentially hundreds of thousands more results to show. This practically controls the time it takes to resolve each instance. Which in turn helps control DDoS attacks by limiting the "damage" each instance can cause.

It also, however, stops real users from seeing all results. It's ultimately a trade-off.

I never suggested to remove the limit, only to filter results at the database level, before applying this limit.

Filter the results with the query, and then truncate, only after having already gotten the most relevant results.

The problem is with that is that it essentially renders your limitation useless, as each query will still access and search through the entire database, leaving it exposed to exploits such as DDoS attacks, while the drawback of having a limit would still apply to users. I doubt that could ever be the plan.

Edit: Didn't scroll down before I posted, basically what Cronus said.