[Davical-general] Hardware sizing

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

[Davical-general] Hardware sizing

Gregory Sloop
I'm looking at deploying DavliCal and I've done some searching about
sizing of a VM or other hardware - but I don't see much [ok, pretty
much anything] in the list archives or the Wiki.

---
Can someone give some very rough guidance on memory and CPU
requirements? [Or point me to a post that discusses this.]

I understand that there are *vast* differences in how the system might
be used that make this difficult, but consider these kinds of setups.

Consider this setup.
---
15-40 users.
Individual calendars, and say 5 shared Cals.
Same with address books. Individuals with say a thousand contacts, and
several "shared" contact lists with the same.

One or two power users in the system which add/change hundreds of
items a day. Others changing a few dozen.

We'll be syncing mobile devices [Android and iOS] every 15m for every
user. We'll also be using TBird caldav/carddav add-ins.

Finally, we'll also be using the CardDavMate/CalDavZAP web front ends.

---
So, to make such a system work well, what kind of CPU and memory
resources should we plan on.

---
I have several options:
VPS with say a GB of memory and reasonable disk-space.
VirtualBox VM with more memory [say up to 2G] and essentially
unlimited disk.
Dedicated hardware. The sky is the limit.

I assume the disk IO is probably the biggest hit, but given the size
I'm talking above, lots of the data can probably be cached in memory
etc.

So, give me some idea.

---
Finally, lets triple/quadruple the size above.
150 users, 10 power users.

How much does this scale the requirements above?
I assume it doesn't triple - but how much.
Again, I assume that disk IO is likely to be the biggest factor, but
even slowish spinning disks should, IMO, handle this quite easily, as
long as they aren't buried by other tasks.

TIA
-Greg



------------------------------------------------------------------------------
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: Hardware sizing

Mateusz Viste
My usage is not far from what you describe (~15 users, most run their
calendars via Thunderbird, 1 or 2 use a CalDavZap frontend).

For me, Davical runs comfortably on a VM with:
  - 8G of disk space
  - 256M of RAM (it never swaps)
  - a single-core CPU @1GHz

Of course YMMV, but I'd say Davical will run happily pretty much on
anything :)

Mateusz



On 04/02/2014 06:43 PM, Gregory Sloop wrote:

> I'm looking at deploying DavliCal and I've done some searching about
> sizing of a VM or other hardware - but I don't see much [ok, pretty
> much anything] in the list archives or the Wiki.
>
> ---
> Can someone give some very rough guidance on memory and CPU
> requirements? [Or point me to a post that discusses this.]
>
> I understand that there are *vast* differences in how the system might
> be used that make this difficult, but consider these kinds of setups.
>
> Consider this setup.
> ---
> 15-40 users.
> Individual calendars, and say 5 shared Cals.
> Same with address books. Individuals with say a thousand contacts, and
> several "shared" contact lists with the same.
>
> One or two power users in the system which add/change hundreds of
> items a day. Others changing a few dozen.
>
> We'll be syncing mobile devices [Android and iOS] every 15m for every
> user. We'll also be using TBird caldav/carddav add-ins.
>
> Finally, we'll also be using the CardDavMate/CalDavZAP web front ends.
>
> ---
> So, to make such a system work well, what kind of CPU and memory
> resources should we plan on.
>
> ---
> I have several options:
> VPS with say a GB of memory and reasonable disk-space.
> VirtualBox VM with more memory [say up to 2G] and essentially
> unlimited disk.
> Dedicated hardware. The sky is the limit.
>
> I assume the disk IO is probably the biggest hit, but given the size
> I'm talking above, lots of the data can probably be cached in memory
> etc.
>
> So, give me some idea.
>
> ---
> Finally, lets triple/quadruple the size above.
> 150 users, 10 power users.
>
> How much does this scale the requirements above?
> I assume it doesn't triple - but how much.
> Again, I assume that disk IO is likely to be the biggest factor, but
> even slowish spinning disks should, IMO, handle this quite easily, as
> long as they aren't buried by other tasks.
>
> TIA
> -Greg
>

------------------------------------------------------------------------------
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: Hardware sizing

Vincent Van Houtte
On 02/04/2014 20:30, Mateusz Viste wrote:

> My usage is not far from what you describe (~15 users, most run their
> calendars via Thunderbird, 1 or 2 use a CalDavZap frontend).
>
> For me, Davical runs comfortably on a VM with:
>   - 8G of disk space
>   - 256M of RAM (it never swaps)
>   - a single-core CPU @1GHz
>
> Of course YMMV, but I'd say Davical will run happily pretty much on
> anything :)

That's quite impressive!

Did you also test with Evolution as a client? My experience with
evolution is that it is *very* slow to display calendar information at
our office ( 7 users, all having both evolution and a mobile client ).
The server is running more than only DAViCal, but if DAViCal is so low
on resources, then I should look for other causes of the slowness...


Tia!

Vincent
--
Vincent Van Houtte
Advocaat
Advocatenkantoor Suy, Van Baeveghem & Van Houtte
Brusselsestraat 108
9200 Dendermonde
T 052 52 06 05
T 052 77 90 05
F 052 52 06 46
W http://synergylaw.be

------------------------------------------------------------------------------
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: Hardware sizing

Mateusz Viste
Nope, never tried Evolution. My understanding is that Thunderbird is not
connected to the calendar server all the time, but only synchronizes the
calendars from time to time. Maybe evolution behaves differently, who
knows..

cheers,
Mateusz



On 04/03/2014 10:00 AM, Vincent Van Houtte wrote:

> On 02/04/2014 20:30, Mateusz Viste wrote:
>> My usage is not far from what you describe (~15 users, most run their
>> calendars via Thunderbird, 1 or 2 use a CalDavZap frontend).
>>
>> For me, Davical runs comfortably on a VM with:
>>    - 8G of disk space
>>    - 256M of RAM (it never swaps)
>>    - a single-core CPU @1GHz
>>
>> Of course YMMV, but I'd say Davical will run happily pretty much on
>> anything :)
>
> That's quite impressive!
>
> Did you also test with Evolution as a client? My experience with
> evolution is that it is *very* slow to display calendar information at
> our office ( 7 users, all having both evolution and a mobile client ).
> The server is running more than only DAViCal, but if DAViCal is so low
> on resources, then I should look for other causes of the slowness...
>
>
> Tia!
>
> Vincent
>

------------------------------------------------------------------------------
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: Hardware sizing

Ján Máté-2
In reply to this post by Vincent Van Houtte
There are *VERY* big differences between DAViCal HW usage with different clients.
Evolution and Lightning are more WebDAV clients than CalDAV clients, and perform
very expensive synchronization using propfind and URL+Etag comparison. In this case
a simple synchronization can eat LOT of bandwidth and also CPU because DAViCal
needs to return the list of all objects (note: NOT content, only URL + Etag) in all synchronized
collection (and create a XML response).

Better clients (iOS, CalDavZAP, CardDavMATE) use "sync-collection report" and instead
of downloading the whole list of objects (URL + Etag) in XML, these clients send the previous
"sync-token" and DAViCal returns the list of CHANGED URLs (if any).

In short:
propfind (for synchronization) = very expensive operation (especially for large collections) = ~O(n)
report (sync-collection) = very simple operation (depends not on the size of the collection but
the number of changed objects since the last synchronization) =  ~O(1)

One of the main reasons why CalDavZAP exists was a POOR performance/behaviour
of Lightning (after switching to CalDavZAP the same HW can serve > 5x more clients with 3 min.
sync interval instead of 5 min. in Lightning) ...


JM


On 03 Apr 2014, at 10:00, Vincent Van Houtte <[hidden email]> wrote:

> On 02/04/2014 20:30, Mateusz Viste wrote:
>> My usage is not far from what you describe (~15 users, most run their
>> calendars via Thunderbird, 1 or 2 use a CalDavZap frontend).
>>
>> For me, Davical runs comfortably on a VM with:
>>  - 8G of disk space
>>  - 256M of RAM (it never swaps)
>>  - a single-core CPU @1GHz
>>
>> Of course YMMV, but I'd say Davical will run happily pretty much on
>> anything :)
>
> That's quite impressive!
>
> Did you also test with Evolution as a client? My experience with
> evolution is that it is *very* slow to display calendar information at
> our office ( 7 users, all having both evolution and a mobile client ).
> The server is running more than only DAViCal, but if DAViCal is so low
> on resources, then I should look for other causes of the slowness...
>
>
> Tia!
>
> Vincent
> --
> Vincent Van Houtte
> Advocaat
> Advocatenkantoor Suy, Van Baeveghem & Van Houtte
> Brusselsestraat 108
> 9200 Dendermonde
> T 052 52 06 05
> T 052 77 90 05
> F 052 52 06 46
> W http://synergylaw.be
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Davical-general mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/davical-general

------------------------------------------------------------------------------

_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

[Davical-general] Hardware sizing

Chris McAnally
In reply to this post by Gregory Sloop
       
Gregory,

Davical doesn¹t have large requirements.  The Hard Disk size depends on
the number of calendars of course, but they normally do not take much
space.  

For our school with over 30 calendars (15 are highly used and packed with
events) we can get by with 20 GB of space (we moved it to a VPS
temporarily while we did upgrades).  We have about 200 devices polling the
calendars over caldav and a large number poll iCal feeds through our web
calendar front-end.

As for RAM, 2GB of RAM has proven more than adequate for us and we also
run our own webcalendar that displays our content on the same server.

I would recommend 2-4 CPU cores, but that¹s general guesswork.  It depends
on the traffic your site gets.

I think you can probably run what you have on the VPS with 1GB of RAM,
maybe up that to 2GB if you really need to.

---
Chris McAnally
Technology Coordinator
SCIS Hongqiao Campus
1161 Hongqiao Road, Shanghai, China 200051
My Site: http://thinkingitthrough.org/
Phone: 86-21-6261-4338 € Fax: 86-21-6261-4639
Cell: 186 2118 7481 € Skype: CheezItMan
Email: [hidden email] € Website: www.scis-his.org
<https://www.scis-his.org/>






On 4/3/14, 4:55 PM, "[hidden email]"
<[hidden email]> wrote:

>[Davical-general] Hardware sizing


------------------------------------------------------------------------------
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: Hardware sizing

Vincent Van Houtte
In reply to this post by Ján Máté-2
On 03/04/2014 10:55, Ján Máté wrote:

> There are *VERY* big differences between DAViCal HW usage with
> different clients.
> Evolution and Lightning are more WebDAV clients than CalDAV clients,
> and perform
> very expensive synchronization using propfind and URL+Etag comparison.
> In this case
> a simple synchronization can eat LOT of bandwidth and also CPU because
> DAViCal
> needs to return the list of all objects (note: NOT content, only URL +
> Etag) in all synchronized
> collection (and create a XML response).
>
> Better clients (iOS, CalDavZAP, CardDavMATE) use "sync-collection
> report" and instead
> of downloading the whole list of objects (URL + Etag) in XML, these
> clients send the previous
> "sync-token" and DAViCal returns the list of CHANGED URLs (if any).
>
> In short:
> propfind (for synchronization) = very expensive operation (especially
> for large collections) = ~O(n)
> report (sync-collection) = very simple operation (depends not on the
> size of the collection but
> the number of changed objects since the last synchronization) =  ~O(1)
>
> One of the main reasons why CalDavZAP exists was a POOR
> performance/behaviour
> of Lightning (after switching to CalDavZAP the same HW can serve > 5x
> more clients with 3 min.
> sync interval instead of 5 min. in Lightning) ...

Interesting to know! I'm quite dissatisfied with Evolution, and this
might be that little 'push over the cliff' for me to schedule in some
time to migrate away from it.

I'll start by having a new look at CalDavZap.

Thx,
Vincent
--
Vincent Van Houtte
Advocaat
Advocatenkantoor Suy, Van Baeveghem & Van Houtte
Brusselsestraat 108
9200 Dendermonde
T 052 52 06 05
T 052 77 90 05
F 052 52 06 46
W http://synergylaw.be

------------------------------------------------------------------------------
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: Hardware sizing

Josef Kufner
On 3.4.2014 11:56, Vincent Van Houtte wrote:
> Interesting to know! I'm quite dissatisfied with Evolution, and this
> might be that little 'push over the cliff' for me to schedule in some
> time to migrate away from it.
>
> I'll start by having a new look at CalDavZap.

Keep in mind, that CalDavZap will never allow you to accept event
invitation from an e-mail and send confirmation back to organizer. That
is way more important than a little bit higher load on server.


------------------------------------------------------------------------------

_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general

signature.asc (898 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Hardware sizing

Ján Máté-2
Hi Josef,

we are working on invitation support (and it is ~ 70% complete). During this
process we found LOT of bugs in DAViCal/Awl (current GIT version) and we will
try to fix them too ... Then the only missing thing will be "external invitation
support" in DAViCal and you will not need to use e-mail client (for invitation)
at all :-)


JM

p.s.: "... more important than a little bit higher load" - O(n) vs O(1) is NOT "little bit"


On 03 Apr 2014, at 12:05, Josef Kufner <[hidden email]> wrote:

> On 3.4.2014 11:56, Vincent Van Houtte wrote:
>> Interesting to know! I'm quite dissatisfied with Evolution, and this
>> might be that little 'push over the cliff' for me to schedule in some
>> time to migrate away from it.
>>
>> I'll start by having a new look at CalDavZap.
>
> Keep in mind, that CalDavZap will never allow you to accept event
> invitation from an e-mail and send confirmation back to organizer. That
> is way more important than a little bit higher load on server.
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Davical-general mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/davical-general

------------------------------------------------------------------------------

_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Hardware sizing

Johan Vromans
In reply to this post by Ján Máté-2
Ján Máté <[hidden email]> writes:

> Better clients (iOS, CalDavZAP, CardDavMATE) use "sync-collection
> report" and instead of downloading the whole list of objects (URL +
> Etag) in XML, these clients send the previous "sync-token" and DAViCal
> returns the list of CHANGED URLs (if any).

However, an unexpected disadvantage of this approach is that when a
client 'misses' a change (yes it happens sometimes!) this will never get
fixed.

In Android I can force a complete restore by uninstalling CalDAVsync and
installing it again, though I wish there was a more elegant way to
completely rebuild the calendars.

-- Johan

------------------------------------------------------------------------------
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: Hardware sizing

Ján Máté-2
:-)))

this must be a bug in your client because the correct synchronization procedure is:

- sync-collection report => get the list of new/updated/deleted data + the new sync token
- multiget report (for the new/update data) => the the new/changed data
- if multiget report failed then perform synchronization with the OLD sync-token (otherwise
perform the new synchronization with the new sync-token)


JM


On 03 Apr 2014, at 12:56, Johan Vromans <[hidden email]> wrote:

> Ján Máté <[hidden email]> writes:
>
>> Better clients (iOS, CalDavZAP, CardDavMATE) use "sync-collection
>> report" and instead of downloading the whole list of objects (URL +
>> Etag) in XML, these clients send the previous "sync-token" and DAViCal
>> returns the list of CHANGED URLs (if any).
>
> However, an unexpected disadvantage of this approach is that when a
> client 'misses' a change (yes it happens sometimes!) this will never get
> fixed.
>
> In Android I can force a complete restore by uninstalling CalDAVsync and
> installing it again, though I wish there was a more elegant way to
> completely rebuild the calendars.
>
> -- Johan
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Davical-general mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/davical-general

------------------------------------------------------------------------------

_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Hardware sizing

Ján Máté-2
/the the/get the/

JM


On 03 Apr 2014, at 13:02, Ján Máté <[hidden email]> wrote:

:-)))

this must be a bug in your client because the correct synchronization procedure is:

- sync-collection report => get the list of new/updated/deleted data + the new sync token
- multiget report (for the new/update data) => the the new/changed data
- if multiget report failed then perform synchronization with the OLD sync-token (otherwise
perform the new synchronization with the new sync-token)


JM


------------------------------------------------------------------------------

_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Hardware sizing

Marten Gajda-2
In reply to this post by Johan Vromans
Am 03.04.2014 12:56, schrieb Johan Vromans:
> Ján Máté <[hidden email]> writes:
>
>> Better clients (iOS, CalDavZAP, CardDavMATE) use "sync-collection
>> report" and instead of downloading the whole list of objects (URL +
>> Etag) in XML, these clients send the previous "sync-token" and DAViCal
>> returns the list of CHANGED URLs (if any).
> However, an unexpected disadvantage of this approach is that when a
> client 'misses' a change (yes it happens sometimes!) this will never get
> fixed.
Actually that should not happen unless either the client or the server
is broken. Have you seen this behavior in CalDAV-Sync on Android?
> In Android I can force a complete restore by uninstalling CalDAVsync and
> installing it again, though I wish there was a more elegant way to
> completely rebuild the calendars.
Since version 0.4 you can enable "Use WebDAV sync method" in the account
setting to switch to the simple PROPFIND method. That will use a
full-sync. Once you disable this option it will return to sync-collection.
However, we've never received any reports about missed changes and we've
never noticed them ourselves. The app will not store the new sync-token
unless the sync has been completed without error.

Marten
>
> -- Johan
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Davical-general mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/davical-general


--
Marten Gajda
Schandauer Straße 34
01309 Dresden
Germany

tel: +49 177 4427167
email: [hidden email]
twitter: twitter.com/dmfs_org

VAT Reg. No.: DE269072391


------------------------------------------------------------------------------
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: Hardware sizing

Marten Gajda
In reply to this post by Johan Vromans
Am 03.04.2014 12:56, schrieb Johan Vromans:
> Ján Máté <[hidden email]> writes:
>
>> Better clients (iOS, CalDavZAP, CardDavMATE) use "sync-collection
>> report" and instead of downloading the whole list of objects (URL +
>> Etag) in XML, these clients send the previous "sync-token" and DAViCal
>> returns the list of CHANGED URLs (if any).
> However, an unexpected disadvantage of this approach is that when a
> client 'misses' a change (yes it happens sometimes!) this will never get
> fixed.
Actually that should not happen unless either the client or the server
is broken. Have you seen this behavior in CalDAV-Sync on Android?
> In Android I can force a complete restore by uninstalling CalDAVsync and
> installing it again, though I wish there was a more elegant way to
> completely rebuild the calendars.
Since version 0.4 you can enable "Use WebDAV sync method" in the account
setting to switch to the simple PROPFIND method. That way the app will
always perform a full-sync. Once you disable this option it will return
to sync-collection.
However, we've never received any reports about missed changes and we've
never noticed them ourselves. The app will not store the new sync-token
unless the sync has been completed without error.

Marten
>
> -- Johan
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Davical-general mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/davical-general


--
Marten Gajda
Schandauer Straße 34
01309 Dresden
Germany

tel: +49 177 4427167
email: [hidden email]
twitter: twitter.com/dmfs_org

VAT Reg. No.: DE269072391

------------------------------------------------------------------------------
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: Hardware sizing

Johan Vromans
In reply to this post by Marten Gajda-2
Marten Gajda <[hidden email]> writes:

> Since version 0.4 you can enable "Use WebDAV sync method" in the
> account setting to switch to the simple PROPFIND method. That will use
> a full-sync. Once you disable this option it will return to
> sync-collection.

Okay, nice to know.

> However, we've never received any reports about missed changes and
> we've never noticed them ourselves. The app will not store the new
> sync-token unless the sync has been completed without error.

I have had several problems with duplicate and corrupted events which I
thought were fixed now. There may have been some remaining problems,
though.

I'll keep a sharp eye to see if it happens again, and whether this
happens to new or old events. I have logging enabled in CalDAVsync but
often I'm just too late to intercept the log.

So just assume there are no problems on your side until I can prove
otherwise.

Thanks for the good work and the on-going support. This is what makes
the world go round.

-- Johan

------------------------------------------------------------------------------
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general